JP2017102564A - Display control program, display control method and display control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display control program that can grasp an attention degree with respect to a plurality of products displayed in a display area of a store and an operation degree with respect to the product.SOLUTION: A display control program is configured to make a computer execute processing of: acquiring behavior information on purchasers in stores, and visual line information on the purchaser relative to a display area of the store having a plurality of products of sales objects displayed; displaying a heat map indicative of an attention degree of the purchaser relative to a location in association with a location in the display area on a display part on the basis of the behavior information, the visual line information, and information stored in a storage unit for specifying a display location of the plurality of products in the display area; and displaying graphics or characters in which a display size is determined in accordance with a behavior degree of the purchaser relative to each of the plurality of products in a location corresponding to each of the plurality of products with the plurality of products overlapped on the heat map.SELECTED DRAWING: Figure 21

Description

  This case relates to a display control program, a display control method, and a display control apparatus.

  As a technique for analyzing the behavior of customers in a store, there is known a technology capable of calculating statistics regarding in-store behavior for each purchaser and non-purchase customer (see, for example, Patent Document 1).

International Publication No. 2015/025490

  By the way, if a retailer or manufacturer can grasp the product that the customer has been paying attention to and the action taken for that product when purchasing the product displayed in the display area such as a display shelf or a wagon in the store, the customer will be temporarily Even if they do not purchase the product, there are significant advantages to retail sales strategies and manufacturer product development.

  Therefore, in one aspect, an object is to provide a display control program, a display control method, and a display control device that can grasp the degree of attention to a plurality of products displayed in a display area of a store and the degree of operation on the products. To do.

  The display control program disclosed in the present specification acquires, in a computer, operation information of a purchaser in a store and line-of-purchase information of the purchaser with respect to a display area of the store displaying a plurality of products to be sold, Based on movement information, the line-of-sight information, and information for specifying display positions of the plurality of products in the display area stored in a storage unit, the position relative to the position is associated with the position in the display area A graphic or character that displays a heat map indicating the degree of attention of the buyer on the display unit and that is superimposed on the heat map and whose display size is determined according to the degree of purchaser's action on each of the plurality of products. Is displayed at a position corresponding to each of the plurality of products in the heat map. That.

  The display control program disclosed in the present specification acquires, in a computer, operation information of a purchaser in a store and line-of-purchase information of the purchaser with respect to a display area of the store displaying a plurality of products to be sold, Based on movement information, the line-of-sight information, and information for specifying display positions of the plurality of products in the display area stored in a storage unit, the position relative to the position is associated with the position in the display area While displaying a heat map indicating the degree of buyer's attention on the display unit and superimposing it on the heat map, a figure whose display size is determined according to the degree of purchaser's action on each of the plurality of products and the figure Characters whose display size is determined in advance regardless of the degree of purchaser's movement are displayed at positions corresponding to the plurality of products in the heat map. To a display control program for causing to execute a process.

  In the display control method disclosed in the present specification, the computer acquires the purchaser's operation information in the store, and the purchaser's line-of-sight information for the display area of the store displaying a plurality of products to be sold, Based on movement information, the line-of-sight information, and information for specifying display positions of the plurality of products in the display area stored in a storage unit, the position relative to the position is associated with the position in the display area A graphic or character that displays a heat map indicating the degree of attention of the buyer on the display unit and that is superimposed on the heat map and whose display size is determined according to the degree of purchaser's action on each of the plurality of products. Is displayed at a position corresponding to each of the plurality of products in the heat map, and the process is executed.

  In the display control method disclosed in the present specification, the computer acquires the purchaser's operation information in the store, and the purchaser's line-of-sight information for the display area of the store displaying a plurality of products to be sold, Based on movement information, the line-of-sight information, and information for specifying display positions of the plurality of products in the display area stored in a storage unit, the position relative to the position is associated with the position in the display area While displaying a heat map indicating the degree of buyer's attention on the display unit and superimposing it on the heat map, a figure whose display size is determined according to the degree of purchaser's action on each of the plurality of products and the figure Characters whose display size is determined in advance regardless of the degree of purchaser's movement are displayed at positions corresponding to the plurality of products in the heat map. A display control method and executes the processing.

  The display control device disclosed in the present specification, the information acquisition unit for acquiring the purchaser's operation information in the store, and the purchaser's line-of-sight information for the display area of the store displaying a plurality of products to be sold, Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, and that is superimposed on the heat map, and whose display size is determined according to the degree of purchaser's action on each of the plurality of products, or An information processing unit that displays characters at positions corresponding to each of the plurality of products in the heat map.

  The display control device disclosed in the present specification, the information acquisition unit for acquiring the purchaser's operation information in the store, and the purchaser's line-of-sight information for the display area of the store displaying a plurality of products to be sold, Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, is superimposed on the heat map, and a display size is determined according to the degree of the purchaser's action on each of the plurality of products, and Information for displaying a character whose display size is determined in advance regardless of the degree of movement of the purchaser at a position corresponding to each of the plurality of products in the heat map. A processing unit, a display control device characterized by having a.

  According to the display control program, the display control method, and the display control device disclosed in the present specification, it is possible to grasp the degree of attention to a plurality of products displayed in the display area of the store and the degree of operation with respect to the products.

FIG. 1 is an example of a purchasing behavior analysis system. FIG. 2 is a diagram for explaining an example of a display shelf. FIG. 3 is an example of a hardware configuration of the display control apparatus. FIG. 4 is an example of a functional block diagram of the display control device. FIG. 5 is a flowchart illustrating an example of information processing executed by the display control apparatus. FIG. 6 is a flowchart illustrating an example of the teacher data generation process. FIG. 7 is a diagram for explaining an example of how teacher data is generated. FIG. 8 is a diagram for explaining an example of how teacher data is generated. FIG. 9 is an example of flow line coordinate data. FIG. 10 is an example of flow line coordinate data associated with POS information. FIG. 11 is a diagram for explaining a processing example of the flow line coordinate data. FIG. 12 is a diagram for explaining a processing example of the line-of-sight coordinate data. FIG. 13 is a diagram for explaining the association between post-process flow line data and post-process line-of-sight data. FIG. 14 is a flowchart illustrating an example of the operation determination process. FIG. 15 is a diagram for explaining an example of the operation determination. FIG. 16 is a diagram for explaining an example of the pre-shelf behavior. FIG. 17 is an example of second behavior data. FIG. 18 is an example of a screen display output in the behavior receipt format. FIG. 19 is an example of a screen display output in the product comparison map format. FIG. 20 is an example of a screen display output in the access analysis format. FIG. 21 is an example of a heat map. FIG. 22 is an example of a graph showing the relationship between the attention level and the number of sales.

  Hereinafter, an embodiment for carrying out this case will be described with reference to the drawings.

  FIG. 1 is an example of a purchasing behavior analysis system S. FIG. 2 is a diagram for explaining an example of the display shelf 10. As shown in FIG. 1, the purchase behavior analysis system S includes wireless sensor cameras 100-1, 100-2, 100-3, a receiver 100-4, line-of-sight sensors 120-1, 120-2,. The display control device 200 is included as a component. The display control device 200 is connected to the receiver 100-4, the line-of-sight sensors 120-1, 120-2,..., The input device 710, and the display device 720. In the present embodiment, as shown in FIG. 1, a convenience store will be described as an example of a retail store. However, the store is not particularly limited to a convenience store, and may be a drug store, for example.

  Here, as shown in FIG. 1, the passable passages in the store of the convenience store are divided into a plurality of areas in advance. For example, the area ID “A” is assigned to the area in front of the display shelf 10 where the lunch boxes are arranged. Similarly, the area ID “C” is assigned to the area closest to the cash register, and the area ID “G” is assigned to the area in contact with the entrance / exit. The same applies to other areas.

  Hereinafter, each component of the purchase behavior analysis system S will be specifically described.

  The wireless sensor cameras 100-1, 100-2, 100-3 are installed at the ceiling of the store, the height of the wall surface, or the like. In FIG. 1, three wireless sensor cameras 100-1, 100-2, and 100-3 are installed, but the number may be increased or decreased depending on the size of the convenience store. Each of the wireless sensor cameras 100-1, 100-2, and 100-3 captures an image of its own imaging area. For example, the wireless sensor camera 100-1 captures an image of the area to which the area ID “C” is assigned. The wireless sensor camera 100-2 captures images in the areas to which the area IDs “A”, “B”, and “K” are assigned. The wireless sensor camera 100-3 captures a part of the area to which the area IDs “G” and “P” are assigned, and the area to which the area IDs “Q” and “R” are assigned.

  The wireless sensor cameras 100-1, 100-2, and 100-3 all transmit captured images, specifically, color moving images (hereinafter simply referred to as moving images) wirelessly. When the wireless sensor cameras 100-1, 100-2, and 100-3 transmit a moving image, the receiver 100-4 receives the moving image and stores it in the display control apparatus 200.

  The wireless sensor cameras 100-1, 100-2, and 100-3 generate identification information (hereinafter referred to as a flow line ID) to be assigned to a customer when the customer enters the imaging area, and image the moving line ID. The position information obtained based on the moving image, the orientation of the customer's face, and the imaged time are associated with each other. Furthermore, the wireless sensor cameras 100-1, 100-2, and 100-3 determine attributes such as the customer's age and sex based on the captured moving image, and associate the attributes with the flow line ID. The wireless sensor cameras 100-1, 100-2, 100-3 transmit the position information, the direction of the customer's face, the time of imaging, and the flow line ID associated with the attribute to the receiver 100-4 as flow line coordinate data. Thereby, the receiver 100-4 obtains the attribute of the customer who entered the store and the flow line in the store. That is, in FIG. 1, a customer who enters a store from the entrance moves through each area in the order of area IDs “G”, “A”, “B”, “A”, “C”, “G”, and exits from the entrance. The receiver 100-4 obtains the flow line.

  The line-of-sight sensors 120-1, 120-2,... Are attached to the display shelf 10 installed in the store. For example, the line-of-sight sensor 120-1 is attached to the display shelf 10 on which lunch boxes are arranged, and the line-of-sight sensor 120-2 is attached to the display shelf 10 on which breads are arranged. The same applies to other line-of-sight sensors. More specifically, referring to FIG. 2, for example, the line-of-sight sensor 120-1 is attached to the shelf plate 11 of the display shelf 10. In FIG. 2, the line-of-sight sensor 120-1 is attached to one of the shelf boards 11, but the line-of-sight sensor 120-1 may be attached to each of the two or more shelf boards 11.

  Each of the line-of-sight sensors 120-1, 120-2,... Detects the line of sight of the customer standing in front of the display shelf 10 using infrared rays or the like. More specifically, the line-of-sight sensor 120-1 generates identification information (hereinafter referred to as line-of-sight ID) assigned to a customer when the head of the customer enters the detection area. Next, for example, as shown in FIG. 2, when the customer's line of sight V moves between a plurality of products (specifically, lunch boxes, rice balls, etc.), the line-of-sight sensor 120-1 detects the position information of the line of sight according to the movement locus. Is detected. The line-of-sight sensor 120-1 associates the detected position information and the detected time with the generated line-of-sight ID. The line-of-sight sensor 120-1 holds the line-of-sight ID associating the position information and the detected time as line-of-sight coordinate data. The same applies to the line-of-sight sensor 120-2.

  The display control apparatus 200 acquires flow line coordinate data from the receiver 100-4. Further, the display control device 200 acquires line-of-sight coordinate data held by the line-of-sight sensors 120-1, 120-2,. Further, the display control apparatus 200 acquires POS information from the Point Of Sales (POS) aggregation system 20. Although details will be described later, the POS information includes, for example, the purchase time of the product, the product name, the number of purchases, the purchase price, and the like. The display control device 200 associates the acquired flow line coordinate data, line-of-sight coordinate data, and POS information with each other, executes various types of information processing, and generates behavior data representing customer behavior. When the display control device 200 generates the behavior data, it uses the generated behavior data to visualize information that can be confirmed by the user, such as a behavior receipt, a product comparison map, and an access analysis, which will be described later. 721 is displayed. The display control device 200, the input device 710, and the display device 720 are included in the components of the terminal device such as a personal computer (PC). Details of the display control apparatus 200 will be described below.

  Next, the hardware configuration of the display control apparatus 200 will be described with reference to FIG. In addition, the same code | symbol is attached | subjected to the structure similar to the display control apparatus 200, the input device 710, and the display apparatus 720 shown in FIG.

  FIG. 3 is an example of a hardware configuration of the display control device 200. As shown in FIG. 3, the display control device 200 includes at least a central processing unit (CPU) 200A, a random access memory (RAM) 200B, a read only memory (ROM) 200C, and an input / output terminal 200D. The display control device 200 includes at least one of a hard disk drive (HDD) 200E, an input I / F (interface) 200F, an output I / F 200G, an input / output I / F 200H, and a drive device 200I as necessary. Also good. The CPU 200A,..., The drive device 200I are connected to each other by an internal bus 200J. At least the CPU 200A and the RAM 200B cooperate to realize a computer.

An input device 710 is connected to the input I / F 200F. Examples of the input device 710 include a keyboard and a mouse.
A display device 720 is connected to the output I / F 200G. An example of the display device 720 is a liquid crystal display.
A semiconductor memory 730 and line-of-sight sensors 120-1, 120-2,... Are connected to the input / output I / F 200H. Examples of the semiconductor memory 730 include a universal serial bus (USB) memory and a flash memory. The input / output I / F 200 </ b> H reads programs and data stored in the semiconductor memory 730.
The input I / F 200F and the input / output I / F 200H include, for example, a USB port. The output I / F 200G includes a display port, for example.

A portable recording medium 740 is inserted into the drive device 200I. Examples of the portable recording medium 740 include a removable disk such as a Compact Disc (CD) -ROM and a Digital Versatile Disc (DVD). The drive device 200I reads a program and data recorded on the portable recording medium 740.
The receiver 100-4 is connected to the input / output terminal 200D.

  In the above-described RAM 200B, a program stored in the ROM 200C or the HDD 200E is stored by the CPU 200A. In the RAM 200B, the program recorded on the portable recording medium 740 is stored by the CPU 200A. When the stored program is executed by the CPU 200A, various functions to be described later are realized, and various operations to be described later are executed. The program may be in accordance with a flowchart described later.

  Next, each function of the display control apparatus 200 will be described with reference to FIG.

  FIG. 4 is an example of a functional block diagram of the display control device 200. As illustrated in FIG. 4, the display control device 200 includes a moving image storage unit 201, an in-store area storage unit 202, a product arrangement storage unit 203, an information acquisition unit 204, an information processing unit 205, and a behavior data storage unit 206.

The moving image storage unit 201 stores moving images. As described above, the moving image storage unit 201 stores the moving image received by the receiver 100-4. The moving image is transmitted from the wireless sensor cameras 100-1, 100-2, 100-3 to the receiver 100-4 via the wireless network NW.
The in-store area storage unit 202 stores in-store area master data. As will be described later with reference to FIG. 11, the in-store area master data includes an area ID that identifies each area described above, an area name that is the name of each area, upper-left and lower-right position information that identifies each area, Includes the face direction the customer points to within the area.

  The product arrangement storage unit 203 stores product arrangement master data. As will be described later with reference to FIG. 12, the product arrangement master data includes the upper-left and lower-right position information for specifying each area when the display shelf 10 is divided into a plurality of areas, and the products arranged in each area. Is associated with the area ID described above.

  The information acquisition unit 204 acquires the flow line coordinate data of the purchaser in the store from the receiver 100-4. The purchaser is a prospective purchaser, and includes both those who actually purchased and those who did not purchase. Further, the information acquisition unit 204 acquires line-of-sight coordinate data held by the line-of-sight sensors 100-1, 100-2,. Further, the information acquisition unit 204 acquires POS information from the POS aggregation system 20. The information acquisition unit 204 outputs the acquired flow line coordinate data, line-of-sight coordinate data, and POS information to the information processing unit 205.

  When the information processing unit 205 acquires the flow line coordinate data, the line-of-sight coordinate data, and the POS information, the moving image stored in the moving image storage unit 201, the in-store area master data stored in the in-store area storage unit 202, and the product arrangement storage unit The product arrangement master data stored in 203 is extracted. The information processing unit 205 executes various types of information processing (data processing) using flow line coordinate data, line-of-sight coordinate data, POS information, moving images, in-store area master data, and product arrangement master data. For example, the information processing unit 205 associates (associates) the flow line coordinate data, the line-of-sight coordinate data, and the POS information with each other using the in-store area master data and the product arrangement master data, and generates the first action data. . After that, the information processing unit 205 associates the action in front of the shelf that measures the time of the action performed in front of the display shelf 10 or the number of times with the first action data, and acts as the second action data. The data is stored in the data storage unit 206. As a result, the behavior data storage unit 206 stores second behavior data representing the behavior of the customer in the store.

  In addition, the information processing unit 205 extracts the second behavior data stored in the behavior data storage unit 206, visualizes it as information that can be confirmed by the user, and displays the information on the display unit 721 of the display device 720. For example, when the information processing unit 205 receives selection (for example, purchase) of any one of a plurality of products, the information processing unit 205 determines the product determined from the product placement master data and the in-store area master data based on the second behavior data. A radar chart that refers to classification information (for example, lunch boxes, breads, etc.), and an index indicating the purchaser's behavior and an index indicating the degree of attention to other products in the same classification as any of the selected products The data is displayed on the display unit 721 in a format.

  Next, the operation of the display control apparatus 200 will be described with reference to FIGS.

  FIG. 5 is a flowchart illustrating an example of information processing executed by the display control apparatus 200. FIG. 6 is a flowchart illustrating an example of the teacher data generation process. 7 and 8 are diagrams for explaining an example of how teacher data is generated.

  First, as shown in FIG. 5, the information processing unit 205 executes a teacher data generation process (step S101). More specifically, as illustrated in FIG. 6, the information processing unit 205 deletes the background image from the captured image (step S201). A captured image is prepared in advance. As shown in FIG. 7A, the captured image is a still image (hereinafter referred to as a still image) obtained by capturing an image of a specific imaging region including the person M. The person M may be a customer or a person other than the customer. FIG. 7A illustrates an operation in which the person M picks up the product. Further, as shown in FIG. 7B, a still image obtained by imaging the same specific imaging area not including the person M is also prepared as a background image. In the present embodiment, both the captured image and the background image are images captured by the wireless sensor camera 100-2.

  The information processing unit 205 deletes the background image from the captured image prepared in advance as described above. Thereby, as shown in FIG.7 (c), the difference image from which the background image was deleted from the captured image is produced | generated. The person M remains in the difference image. When the process of step S201 is completed, the information processing unit 205 then monochromeizes the difference image (step S202). Monochrome is also called gray scale, for example. Thereby, as shown in FIG. 8A, the person M included in the difference image is replaced with an image region including at least one of gray and black. In FIG. 8A, the hatched gray color is shown, and the difference in shading is expressed by the thickness of the hatching. As a result, the person M is converted into a format in which the person M cannot be specified while representing the action of the person M. Therefore, the privacy of the person M is protected.

  When the process of step S202 is completed, the information processing unit 205 then converts the monochrome difference image into a CSV file (step S203). More specifically, as shown in FIG. 8B, the information processing unit 205 monochrome scanned differential images are sequentially scanned from the top, and as shown in FIG. 8C, the portion representing white is set to “0”. Then, the CSV file 30 is generated by replacing the part representing gray to black with “real number greater than 0 and less than or equal to 1”. As a result, the monochrome difference image is converted into a CSV file 30 representing an operation of picking up a product. When the process of step S203 is completed, the information processing unit 205 then gives meaning to the CSV file 30 (step S204). Specifically, when the user operates the input device 710 to input the operation content for the CSV file 30, the information processing unit 205, based on the input operation content, as shown in FIG. This means that the CSV file 30 in which “real numbers of 0 or more and 1 or less” are arranged in a specific order represents the operation “take in hand”.

  When the process of step S204 is completed, the information processing unit 205 executes deep learning (deep learning, hereinafter simply referred to as DL) (step S205). Specifically, a large number of non-semantic captured images prepared in advance are converted into CSV files (not shown) by the method described above, and these CSV files are already defined CSV file 30. In contrast to this, meaning is given to each of a large number of CSV files. Therefore, if a large number of unimportant CSV files contain operations that are slightly different from the operation “take”, a large number of unmeaning CSV files are compared with the CSV file 30. As a result, even if they do not completely match, the information processing unit 205 means that a slightly different operation is “pick up”. On the other hand, when a large number of unimportant CSV files include an operation that is significantly different from the operation “take in hand”, when a large number of CSV files and the CSV file 30 are compared, both are large. Since they are different, the information processing unit 205 does not imply the action “pick up” for the action.

  Therefore, in FIGS. 7 and 8, the operation “pick up” has been described as an example, but by the same process, a CSV file having several patterns meaning the operation “see”, the operation “put into the basket”, etc. is also generated. When DL is executed, the operation “view”, the operation “put into the basket”, and the like are also given to the CSV files of the remaining captured images for which the operation “pick up” is not meaningful. As a result, a predetermined number of patterns of operations are implied for CSV files of captured images with various operations. The CSV file to which the action is given is called teacher data.

  Returning to FIG. 5, when the process of step S <b> 101 is completed, the information acquisition unit 204 then acquires flow line coordinate data and line-of-sight coordinate data (step S <b> 102). When the process of step S102 is completed, the information acquisition unit 204 then determines whether there is POS information (step S103). More specifically, the information acquisition unit 204 accesses the POS aggregation system 20 and determines whether POS information is stored in the POS aggregation system 20. When the information acquisition unit 204 determines that the POS terminal 20 has POS information (step S103: YES), the information acquisition unit 204 acquires POS information, and the information processing unit 205 associates the POS information with the flow line coordinate data. (Step S104). On the other hand, when the information acquisition unit 204 determines that there is no POS information in the POS aggregation system 20 (step S103: NO), the process of step S104 is skipped. Further, when the processing in step S103 or S104 is completed, the first action data is generated by associating the flow line coordinate data with the line-of-sight coordinate data (step S105).

  Here, with reference to FIG. 9 to FIG. 13, a process until the POS information is associated with the flow line coordinate data will be specifically described. FIG. 9 is an example of flow line coordinate data. FIG. 10 is an example of flow line coordinate data associated with POS information. FIG. 11 is a diagram for explaining a processing example of the flow line coordinate data. FIG. 12 is a diagram for explaining a processing example of the line-of-sight coordinate data. FIG. 13 is a diagram for explaining the association between post-process flow line data and post-process line-of-sight data. The flow line coordinate data shown in FIG. 9 represents the flow line of the customer described with reference to FIG. The information acquisition unit 204 acquires, for example, flow line coordinate data shown in FIG.

  The information processing unit 205 processes the flow line coordinate data acquired by the information acquisition unit 204. Specifically, first, the information processing unit 205 extracts flow line coordinate data for a certain period before and after the purchase time based on the purchase time recorded in the POS information. For example, as shown in FIG. 10, when the purchase time is 10:11:02, from the flow line coordinate data shown in FIG. 9 to the flow line coordinate data 62 seconds before the purchase time to 3 seconds after the purchase time. Extract the flow line coordinate data. The predetermined period may be determined in advance according to the design. As a result, the flow line coordinate data shown in FIG. 9 is narrowed down to the flow line coordinate data from 10:10:00 to 10:11:05. The information processing unit 205 sets the position coordinates of the area before the POS aggregation system 20, that is, the area (refer to FIG. 1) to which the area ID “C” is assigned, from the flow line coordinate data thus narrowed down. The POS information is associated with the flow line coordinate data. Therefore, as shown in FIG. 10, the POS information is linked to the flow line coordinate data from 10:10:07 to 10:11:05.

  Thereafter, as illustrated in FIG. 11, the information processing unit 205 uses the in-store area master data to process the flow line coordinate data associated with the POS information. Specifically, the information processing unit 205 matches the position coordinates X, Y and the face direction of the flow line coordinate data associated with the POS information with the position coordinates and the face direction registered in the in-store area master data, Processed flow line data is generated that indicates in what area the customer stayed from what time to what time. In FIG. 11, a part of the POS information in the flow line coordinate data associated with the POS information is omitted. The information processing unit 205 temporarily stores the generated post-process flow line data for use in an operation determination process described later.

  On the other hand, as illustrated in FIG. 12, the information processing unit 205 processes the line-of-sight coordinate data using the product arrangement master data. Specifically, the information processing unit 205 matches the position coordinates X and Y of the line-of-sight coordinate data with the position coordinates registered in the product arrangement master data, and indicates which product the customer has been paying attention to in which area from when. Generate post-processing line-of-sight data.

  Then, as illustrated in FIG. 13, the information processing unit 205 associates the post-processing flow line coordinate data with the post-processing line-of-sight coordinate data, and generates action data. Specifically, the information processing unit 205 matches the area ID of the post-process flow line coordinate data with the area ID of the post-process line-of-sight coordinate data, and the start time and end time of the post-process flow line coordinate data and the post-process line-of-sight coordinates. The first action data is generated by matching the time of the data. Thereby, the 1st action data showing the area where the customer stayed, the time when the customer stayed, the product noticed in the area, the actually purchased product, and the like are generated.

  Returning to FIG. 5, when the process of step S <b> 105 is completed, the information processing unit 205 executes an operation determination process (step S <b> 106). More specifically, as shown in FIG. 14, the information processing unit 205 converts a moving image into a still image (step S301). Since the moving image is stored in the moving image storage unit 201, the information processing unit 205 extracts the moving image from the moving image storage unit 201. Here, as described above, the information processing unit 205 holds post-process flow line data. Therefore, as shown in FIG. 15, the information processing unit 205 is the time of the flow line ID that stays in the area in front of the display shelf 10 (for example, the area of the area ID “A”) for a certain time (for example, 2 seconds). A band is specified, and a moving image is extracted from the moving image storage unit 201 for a moving image in the specified time period. Then, the information processing unit 205 converts the extracted moving image into a still image in units of one second. Thereby, a plurality of still images are generated from the extracted moving image.

  When the processing in step S301 is completed, the information processing unit 205 then deletes the background image from the still image (step S302), and converts the still image from which the background image has been deleted to monochrome (step S303). When the processing in step S303 is completed, the information processing unit 205 then converts the monochrome still image into a CSV file (step S304). When the process of step S304 is completed, the information processing unit 205 then executes DL (step S305). The DL executed in the process of step S305 is an operation determination, and is performed using the teacher data generated in the process of step S101. That is, meaning is not yet performed on the CSV file obtained by the process of step S304.

  On the other hand, the teacher data generated by the process of step S101 is a CSV file that has been given meaning. Therefore, when the information processing unit 205 executes DL using the teacher data, meaning is given to a CSV file that has not yet been made meaningful. Thereby, as shown in FIG. 15, meaning is given to each still image. In other words, the still image at the time 10:10:00 means the operation “see” as a result of the operation determination, and the still images at the time 10:10:01 and 02 seconds have the operation “ "Take it in hand" is meant.

  Further, as shown in FIG. 16, the information processing unit 205 associates the image determination result determined for the operation with a part of the processed flow line data subjected to the operation determination, and measures the decision speed and the attention time. And count the number of picks and attentions. The decision speed is the time from the start of the operation “watch” to the start of the operation “pick up”. Therefore, in this embodiment, as shown in FIG. 16, the information processing unit 205 measures the decision speed as 2 seconds. Further, the information processing unit 205 measures the attention time corresponding to the operation “view” as 1 second. On the other hand, the information processing unit 205 confirms the contents registered in the operation determination in the image determination result, and counts as one operation if the same operation is continuous, and counts each as one operation if the same operation is discontinuous. Count as an action. Therefore, in the present embodiment, as illustrated in FIG. 16, the information processing unit 205 counts the number of times it is picked up as one. In addition, the information processing unit 205 measures the number of attentions corresponding to the operation “view” as one.

  When the information processing unit 205 finishes the measurement and counting, as shown in FIG. 16, the action in front of the shelf, which is the action in front of the display shelf 10, that is, the decision speed, the attention time, the number of times taken, and the attention time Is linked to the flow line data after processing. Furthermore, the information processing unit 205 associates the action in front of the shelf with the action data generated in the process of step S105. The information processing unit 205 includes a flow line ID, area ID, start time and flow line ID of the first action data (see FIG. 13) of post-processing flow line data (see FIG. 16) associated with the action in front of the shelf. The area ID and time are matched, and the action in front of the shelf is associated with the matched action data and stored in the action data storage unit 106. Thereby, as shown in FIG. 17, the behavior data storage unit 106 stores the second behavior data in which the pre-shelf behavior is associated. The second action data is an example and does not represent all the output formats described later.

  Returning to FIG. 5, when the process of step S <b> 106 is completed, the information processing unit 205 visualizes and displays it on the display unit 721 (step S <b> 107). That is, the information processing unit 205 extracts the second behavior data from the behavior data storage unit 206, and converts the second behavior data into various output formats in which a user (for example, a retail store or a person in charge of a manufacturer) can easily understand customer behavior. Are displayed on the display unit 721.

  Hereinafter, specific examples of various output formats will be described with reference to FIGS.

  FIG. 18 is an example of a screen display output in the behavior receipt format. As illustrated in FIG. 18, the information processing unit 205 displays a behavior receipt on the display unit 721. The behavior receipt is an output result representing the behavior of the customer who has moved through the store and performed various operations. The content of the behavior data is displayed on the behavior receipt in time series. For example, if the product “Yakiniku bento” is actually purchased, purchase, purchase amount “580 yen” and decision speed “2 seconds”, which indicate purchase, are displayed. In addition, if the product “cream bread” or “rice ball” that has not been purchased is displayed, it is displayed that the product is merely a product of interest. You may display attention time and the frequency | count of attention by design.

  In addition, since the action data does not have a purchase price for the product of interest, for example, the unit price of each product is linked in advance to the product arrangement master data, and the unit price linked to each product in the product arrangement master data is displayed. You just have to do it. The screen display in the behavior receipt format allows the user to grasp the operation for another product and the degree of attention to the product compared when purchasing a certain product.

  FIG. 19 is an example of a screen display output in the product comparison map format. As illustrated in FIG. 19, the information processing unit 205 displays a product comparison map on the display unit 721. The product comparison map shows an index indicating the purchaser's behavior and an index indicating the degree of attention in the radar chart format for each operation for other products in the same category as the product selected and purchased from multiple products. This is the output result.

  For example, if the product “Yakiniku bento” is actually purchased, the purchased product “Yakiniku bento” is displayed on the radar chart and compared with the category “Bento” to which the product “Yakiniku bento” belongs. The product name of another product and the degree of comparison or comparison time for the product are displayed. Specifically, if a customer compares the product “Chicken lunch” when purchasing the product “Yakiniku bento”, it indicates that the attention corresponding to the action to watch was 2 seconds or 2 times. , Indicates that the action taken by the hand was 5 seconds or 5 times. Note that it is only necessary to determine in accordance with the design whether the comparison degree or the comparison time is displayed. For example, a selection button for selecting a display format may be provided in any region of the product comparison map.

  The screen display in the product comparison map format allows the user to grasp the operation for another product and the degree of attention to the product compared when purchasing a product. For example, if the area representing attention is small, that is, the line of sight is directed to various products, but the time when one product is focused is short, the area representing the action taken by the hand is small, ie, the time taken by the hand is short Therefore, it can be determined that the time until the purchase of the product is decided was fast. That is, it can be analyzed that the customer has entered the store with the purpose, discovered the target product, and has made a purchase.

  FIG. 20 is an example of a screen display output in the access analysis format. FIG. 21 is an example of a heat map. FIG. 22 is an example of a graph showing the relationship between the attention level and the number of sales. As illustrated in FIGS. 20 to 22, the information processing unit 205 displays an access analysis, a heat map, and a graph on the display unit 721, respectively. As shown in FIG. 20, in the access analysis, the pattern of the purchaser's movement in the store specified based on the second behavior data is determined based on the type of product (for example, lunch box) And a tree diagram (tree diagram) that can be selected. Thereby, it is possible to grasp the product type having a high purpose awareness and a high degree of attention for the customer. In FIG. 20, for example, when there are 100 customers who have entered the store, first, 50 people went to the product type “lunch” and 20 people went to the consumption type “magazines”. The information processing unit 205 displays the access analysis of the selected time zone, gender, and age.

  When one of the product types is selected from the product types included in the displayed pattern, the information processing unit 205 moves the selected product type including the selected product type for the product included in the selected product type. Displays detailed information when stopping by a pattern. The detailed information includes, for example, a heat map shown in FIG. 21 and a graph shown in FIG. For example, as illustrated in FIG. 20, when the product type “bento” corresponding to the area visited second by the pointer Pt is selected, the information processing unit 205 displays the heat map HM illustrated in FIG. 21.

  The heat map HM indicates the degree of attention of the customer as a purchaser with respect to the position in association with the position in each area of the display shelf 10. In addition, in the heat map HM, graphics or characters whose display size is determined according to the degree of purchaser's action on each of the plurality of products are superimposed at positions corresponding to the plurality of products in the heat map HM. . In the heat map HM, a graphic whose display size is determined in accordance with the degree of purchaser's movement for each of a plurality of products and a character whose display size is determined in advance regardless of the degree of purchaser's movement are the heat map HM. May be superimposed at positions corresponding to each of a plurality of products. In this embodiment, a circle is used as an example of a figure. The above-described movement pattern is displayed in a display area different from the display area where the heat map HM is displayed. Therefore, in this embodiment, after entering the store and staying in the area of the area ID “Q” or “R” where the magazines are arranged, the customers who come to the area of the area ID “A” where the lunch boxes are arranged The movement pattern of the customer is shown, and the degree of attention of the customer who has come to the lunch box area in such a pattern is shown by the heat map HM.

  For example, the heat map HM indicates that the degree of attention to the purchased product “Yakiniku bento” is higher than other products such as “Chicken lunch box”. It is indicated by the size of the circle CR. The screen display in the heat map format allows the user to grasp the operation for another product and the degree of attention to the product compared when purchasing a certain product.

  Here, when the screen switching button Bt provided in the display area different from the display area where the heat map HM is displayed and the display area where the movement pattern is displayed is pressed by the pointer Pt, the information processing unit 205 22 is displayed. As described above, the graph of the attention level and the sales volume is displayed, so that the user can analyze the correlation between the attention level and the sales volume. When the screen switching button Bt ′ provided in the display area different from the display area of the graph part shown in FIG. 22 is pressed by the pointer Pt, the information processing unit 205 displays the heat map again.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to the specific embodiments according to the present invention, and various modifications are possible within the scope of the gist of the present invention described in the claims.・ Change is possible. For example, in the above-described embodiment, the wireless sensor cameras 120-1, 120-2, and 120-3 are used, but the communication format is not limited to wireless and may be wired. Further, the terminal device including the display control device 200, the input device 710, and the display device 720 has been described as a stationary type (desktop type) as an example, but may be a notebook type. Furthermore, in order to improve the accuracy of the operation determination, a process for removing noise may be included immediately after performing monochrome.

In addition, the following additional notes are disclosed regarding the above description.
(Supplementary Note 1) Acquires, in a computer, operation information of a purchaser in a store and line-of-sight information of the purchaser with respect to a display area of the store displaying a plurality of products to be sold, and the movement information and the line-of-sight information And the information specifying the display positions of the plurality of products in the display area stored in the storage unit, the degree of the buyer's attention to the position in association with the position in the display area A heat map to be displayed is displayed on a display unit, and a graphic or a character whose display size is determined in accordance with a degree of a purchaser's operation with respect to each of the plurality of products is superimposed on the heat map, and the heat map A display control program for executing processing to display at a position corresponding to each of a plurality of products.
(Supplementary Note 2) Acquires, in a computer, operation information of a purchaser at a store and line-of-sight information of the purchaser with respect to a display area of the store displaying a plurality of products to be sold, and the movement information and the line-of-sight information And the information specifying the display positions of the plurality of products in the display area stored in the storage unit, the degree of the buyer's attention to the position in association with the position in the display area A heat map to be displayed is displayed on the display unit, and is superimposed on the heat map so that a display size is determined according to the degree of the purchaser's action on each of the plurality of products and the degree of the purchaser's action Regardless of which the character whose display size is determined in advance is displayed at a position corresponding to each of the plurality of products in the heat map. Display control program which is characterized.
(Additional remark 3) The said process displays the said figure or the said character while displaying the said heat map for every pattern of the said purchaser's movement specified based on the said operation information. The display control program according to 2.
(Supplementary note 4) The display control program according to any one of supplementary notes 1 to 3, wherein the character is a product name of each of the plurality of products.
(Additional remark 5) A computer acquires the purchaser's movement information in a store, and the buyer's line-of-sight information with respect to the display area of the store which displays a plurality of goods for sale, the movement information, and the line-of-sight information And the information specifying the display positions of the plurality of products in the display area stored in the storage unit, the degree of the buyer's attention to the position in association with the position in the display area A heat map to be displayed is displayed on a display unit, and a graphic or a character whose display size is determined in accordance with a degree of a purchaser's operation with respect to each of the plurality of products is superimposed on the heat map, and the heat map A display control method characterized by executing processing to display at a position corresponding to each of a plurality of products.
(Additional remark 6) A computer acquires the purchaser's operation | movement information in a store, and the said buyer's eyes | visual_axis information with respect to the display area of the said store which displays the some goods for sale, The said operation information and the said eyes | visual_axis information And the information specifying the display positions of the plurality of products in the display area stored in the storage unit, the degree of the buyer's attention to the position in association with the position in the display area A heat map to be displayed is displayed on the display unit, and is superimposed on the heat map so that a display size is determined according to the degree of the purchaser's action on each of the plurality of products and the degree of the purchaser's action Regardless of which the display size is determined in advance is displayed at a position corresponding to each of the plurality of products in the heat map. Display control method for the butterflies.
(Supplementary Note 7) The processing includes displaying the heat map for each purchaser movement pattern specified based on the operation information, and displaying the graphic or the character. 6. The display control method according to 6.
(Supplementary note 8) The display control method according to any one of supplementary notes 5 to 7, wherein the character is a product name of each of the plurality of products.
(Additional remark 9) The information acquisition part which acquires the purchaser's operation information in a store, and the buyer's line-of-sight information with respect to the display field of the store which displays a plurality of goods for sale, the operation information, and the line-of-sight Based on the information and the information specifying the display positions of the plurality of products in the display area stored in the storage unit, the degree of the buyer's attention to the position in association with the position in the display area A graphic or a character whose display size is determined in accordance with the degree of the purchaser's action on each of the plurality of products is superimposed on the heat map and displayed on the display unit. An information processing unit that displays information at a position corresponding to each of the plurality of products.
(Additional remark 10) The information acquisition part which acquires the buyer's operation information in a store, and the buyer's gaze information to the display field of the store which displays a plurality of goods for sale, the operation information, and the gaze Based on the information and the information specifying the display positions of the plurality of products in the display area stored in the storage unit, the degree of the buyer's attention to the position in association with the position in the display area A graphic indicating the display size is determined according to the degree of the purchaser's action on each of the plurality of products, and the degree of the purchaser's action Regardless of whether the display size is determined in advance, the information processing unit for displaying the character at a position corresponding to each of the plurality of products in the heat map Display control device according to claim and.
(Additional remark 11) The said information processing part displays the said figure or the said character while displaying the said heat map for every pattern of the said purchaser's movement specified based on the said operation information. The display control apparatus according to 9 or 10.
(Supplementary note 12) The display control device according to any one of supplementary notes 9 to 11, wherein the character is a product name of each of the plurality of products.

S purchase behavior analysis system 100-1, 100-2, 100-3 wireless sensor camera 100-4 receiver 120-1, 120-2 line-of-sight sensor 200 display control device 201 moving image storage unit 202 in-store area storage unit 203 commodity arrangement storage Unit 204 information acquisition unit 205 information processing unit 206 behavior data storage unit 710 input device 720 display device 721 display unit

Claims (8)

On the computer,
Acquiring the purchaser's operation information in the store and the line-of-sight information of the buyer with respect to the display area of the store displaying a plurality of products to be sold;
Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, and that is superimposed on the heat map, and whose display size is determined according to the degree of purchaser's action on each of the plurality of products, or Displaying characters at positions corresponding to each of the plurality of products in the heat map;
A display control program for executing processing. On the computer,
Acquiring the purchaser's operation information in the store and the line-of-sight information of the buyer with respect to the display area of the store displaying a plurality of products to be sold;
Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, is superimposed on the heat map, and a display size is determined according to the degree of the purchaser's action on each of the plurality of products, and Characters whose display size is determined in advance regardless of the degree of the purchaser's action are displayed at positions corresponding to the plurality of products in the heat map,
A display control program for executing processing.   The said process displays the said figure or the said character while displaying the said heat map for every pattern of the said purchaser's movement specified based on the said operation information. Display control program.   The display control program according to any one of claims 1 to 3, wherein the character is a product name of each of the plurality of products. Computer
Acquiring the purchaser's operation information in the store and the line-of-sight information of the buyer with respect to the display area of the store displaying a plurality of products to be sold;
Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, and that is superimposed on the heat map, and whose display size is determined according to the degree of purchaser's action on each of the plurality of products, or Displaying characters at positions corresponding to each of the plurality of products in the heat map;
A display control method characterized by executing processing. Computer
Acquiring the purchaser's operation information in the store and the line-of-sight information of the buyer with respect to the display area of the store displaying a plurality of products to be sold;
Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, is superimposed on the heat map, and a display size is determined according to the degree of the purchaser's action on each of the plurality of products, and Characters whose display size is determined in advance regardless of the degree of the purchaser's action are displayed at positions corresponding to the plurality of products in the heat map,
A display control method characterized by executing processing. An information acquisition unit that acquires the purchaser's operation information in the store and the purchaser's line-of-sight information for the display area of the store that displays a plurality of products to be sold;
Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, and that is superimposed on the heat map, and whose display size is determined according to the degree of purchaser's action on each of the plurality of products, or An information processing unit that displays characters at positions corresponding to each of the plurality of products in the heat map;
A display control device comprising: An information acquisition unit that acquires the purchaser's operation information in the store and the purchaser's line-of-sight information for the display area of the store that displays a plurality of products to be sold;
Based on the motion information, the line-of-sight information, and information specifying the display positions of the plurality of products in the display area stored in the storage unit, the position is associated with the position in the display area A graphic that displays a heat map indicating the degree of attention of the purchaser on a display unit, is superimposed on the heat map, and a display size is determined according to the degree of the purchaser's action on each of the plurality of products, and An information processing unit that displays characters whose display size is determined in advance regardless of the degree of the purchaser's action at a position corresponding to each of the plurality of products in the heat map;
A display control device comprising:

Images