JP2018206354A - Visual line analysis program, visual line analysis method, and visual line analysis device - Google Patents

Abstract

To correctly specify an object that a user has hesitated to select by using visual line direction.SOLUTION: There is provided a visual line analysis program for causing a computer to execute processing for, when detecting a selection event for a first object among a plurality of objects, specifying a second object different from the first object on the basis of a detection state of a visual line about the plurality of objects in a time zone closest to the selection event, and for outputting information indicating the specified second object.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a line-of-sight analysis program, a line-of-sight analysis method, and a line-of-sight analysis apparatus.

  It has been proposed to count the number of products that the user has watched from the start to the end of the purchase behavior of the user, and to estimate the product as a product of which the number of times has been fixed (see, for example, Patent Document 1). ). In Patent Document 1, it is determined that the user has some interest or is wondering whether to purchase the product for all products for which the number of gazes is a certain number or more.

JP 2009-42956 A

  However, in Patent Document 1, it is considered that the meaning of gazing at a product may differ depending on the timing at which the user gazed at the product after starting selection of the product until the selection is completed. Absent. For this reason, there is a problem in that it is erroneously determined that an object that has not been selected is lost.

  Therefore, in one aspect, an object of the present invention is to correctly specify an object for which a user is unsure of selection using line-of-sight information.

  In one embodiment, when a selection event for the first object among a plurality of objects is detected, based on the gaze detection status of the plurality of objects in a time zone closest to the selection event, A line-of-sight analysis program characterized in that a second object different from the first object is specified, information indicating the specified second object is output, and processing is executed by a computer is provided. The

  In one aspect, the present invention can correctly identify an object for which the user has made a selection using line-of-sight information.

The figure explaining the relationship between eyes | visual_axis and selection of goods. The figure which shows an example of a function structure of the visual axis analyzer which concerns on 1st-5th embodiment. The figure which shows an example of the gaze information DB which concerns on one Embodiment, and a product determination result. The figure which shows an example of the hardware constitutions of the gaze analysis apparatus which concerns on one Embodiment. The flowchart which shows an example of the visual line analysis process which concerns on 1st Embodiment. The figure explaining the visual line analysis process which concerns on 1st Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 2nd Embodiment. The figure which shows an example of the method of determining a threshold value in the visual line analysis process which concerns on 2nd Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 3rd Embodiment. The figure explaining the gaze analysis process which concerns on 3rd Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on the modification of 3rd Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 4th Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 5th Embodiment. The figure explaining the gaze analysis processing concerning a 5th embodiment. The figure explaining the gaze analysis processing concerning a 5th embodiment. The figure explaining the gaze analysis processing concerning a 5th embodiment. The figure which shows an example of a function structure of the visual axis analyzer which concerns on 6th-8th embodiment. The figure which shows an example of target object arrangement | positioning DB which concerns on one Embodiment. The figure which shows an example of the lost target DB which concerns on one Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 6th Embodiment. The figure explaining the gaze analysis processing concerning a 6th embodiment. The figure which shows an example of the method of determining a threshold value in the visual line analysis process which concerns on 6th Embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 7th Embodiment. The figure explaining the gaze analysis processing concerning a 7th embodiment. The flowchart which shows an example of the gaze analysis process which concerns on 8th Embodiment. The figure explaining the gaze analysis processing concerning an 8th embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Introduction]
“Gaze” means that the line of sight stays in a predetermined plane area for a predetermined time or longer. The gaze product can be used for sales promotion of the product if the gaze product can be correctly estimated because the user is considered to have some interest or purchase or selection of the product.

  For example, as shown in FIG. 1A, the number of products A to D in which order the user gazes between the start of product selection and the completion of selection, and the number of times of gazing is equal to or greater than a threshold value. Is estimated to be a watched product, all the products A to D shown in FIG.

  However, as shown in FIG. 1C, after looking at a plurality of products in order, the person narrows down the products he / she wants to buy and then sends his / her line of sight as to which of the narrowed products should be purchased. That is, the user frequently views the lost object frequently at a time close to completion of selection. Therefore, if a product whose number of gazes is uniformly equal to or greater than the threshold value is estimated as a gaze product, an object that is not sure to buy is erroneously determined to be lost.

  Therefore, in the line-of-sight analysis apparatus according to the present embodiment, as shown in FIG. 1 (d), the product is a product that is lost in purchase or selection based on an increase amount with the lapse of time of the number of times the user has viewed the object. Whether the product is estimated to be lost or not is output. In this way, based on the line-of-sight data, by correctly estimating the target for which purchase or selection has been lost, it is possible to promote sales of the product based on the information of the product that has been closely watched even if it is a non-purchased product. Can be used in the marketing field. Details of the line-of-sight analysis apparatus according to the present embodiment will be described below.

[Functional structure of eye-gaze analyzer]
First, an example of a functional configuration of the line-of-sight analysis apparatus 10 according to an embodiment of the present invention will be described with reference to FIG. FIG. 2 shows an example of a functional configuration of the visual line analyzer 10 according to the first to fifth embodiments. The line-of-sight analysis apparatus 10 displays information on the line of sight poured into a plurality of products (here, products A to D) drawn on the screen of the digital signage 20 and a plurality of products displayed in the store. It acquires using the gaze sensor 22 attached to the above. The line-of-sight analysis apparatus 10 estimates a “lost product” using the acquired line-of-sight data in a scene where the user selects and views the products on the digital signage 20 or the display shelf.

  The line-of-sight sensor 22 is arranged in front of each shelf in the store or at a position facing the person who views the digital signage 20 and measures the movement of the line of sight of the person. An example of the line-of-sight sensor 22 is a line-of-sight detection camera, but is not limited thereto. For example, as shown in Japanese Patent Application Laid-Open No. 2015-192343, the movement of the line of sight can be acquired using a known line-of-sight sensor. In the present embodiment, the number of line-of-sight sensors 22 is one. However, the present invention is not limited to this, and one or a plurality of line-of-sight sensors 22 may be installed in the digital signage 20 or the store.

  An example of gaze detection using a gaze detection camera is the method of using the positional relationship with the reference point as the eye and the moving point as the iris, and the method of using the positional relationship with the corneal reflection as the reference point and the moving point as the pupil. It is done. In the method of using the positional relationship with the reference point as the head and the moving point as the iris, the gaze detection camera captures the user's movement with a camera that captures visible light. In the method of using the positional relationship with the corneal reflection as the reference point and the moving point as the pupil, the user's face is illuminated with an infrared LED, and the user's eyes are photographed with an infrared camera. The line-of-sight analysis apparatus 10 performs image processing on the image captured by the user's eyes sent from the line-of-sight sensor 22, detects the movement of the eyes, and analyzes the position (plane coordinates) that the user is looking at.

  The line-of-sight analysis apparatus 10 includes a storage unit 11, a detection unit 12, an acquisition unit 13, a determination unit 14, a calculation unit 15, a specification unit 16, and an output unit 17. The storage unit 11 stores programs and data including a line-of-sight information DB 30, a threshold table 31, and a line-of-sight analysis program 32.

  FIG. 3A shows an example of the line-of-sight information DB 30. The line-of-sight information DB 30 associates line-of-sight data when the user directs his / her line of sight to the product displayed on the digital signage 20 or the product displayed in the store, that is, the coordinates (x, y) of the line of sight to the recording time. And save. For example, line-of-sight data about 10 times per second may be recorded. The line-of-sight data may be a pixel unit of the digital signage 20 or a mm unit.

  The threshold value table 31 stores threshold values that are used to determine whether or not a product is watched using line-of-sight data.

  Returning to FIG. 2, the detection unit 12 detects a selection event for a certain object (hereinafter referred to as a “first object”) among a plurality of objects. As an example of detection of the selection event for the first object, detection of selecting any of the products A to D displayed on the digital signage 20 and pressing the selection button 24 may be mentioned. As another example of the detection of the selection event for the first object, there is a detection that a product displayed in a store is registered in a POS system (point of sales system) when the user purchases it. .

  When the line-of-sight analysis program 32 detects the selection event, the line-of-sight analysis program 32 loses the second object that is different from the first object based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. It is a program for specifying the product. Examples of the first object and the second object include products existing in real space, display of products in a virtual space, display of thumbnail images of products, partial enlargement display of products, display of characteristic parts of products, and the like. Can be mentioned.

  Note that the “lost product” described below may be a product selected or purchased in doubt, or may be a product that has been selected but not purchased.

The acquisition unit 13 acquires an image including the state of the user's line of sight from the line-of-sight sensor 22. The determination unit 14 analyzes the acquired image, extracts coordinates to which the user's line of sight is directed, and determines a product viewed by the user from an area including the coordinates of the line of sight. For example, as illustrated in FIG. 3B, the determination unit 14 defines the region of the product A by upper left coordinates (X ₁ , Y ₁ ) and lower right coordinates (X ₂ , Y ₂ ). In this way, from the area information defined for each product and the coordinates (t, x, y) of the line-of-sight data at time t stored in the line-of-sight information DB 30, it is determined which product the user was viewing at time t. It memorize | stores in eyes | visual_axis information DB30. FIG. 3C shows an example in which the product determined to be viewed is stored in the “product viewed” column of the line-of-sight information DB 30.

  The calculation unit 15 calculates the number of times the user has viewed each product at predetermined time intervals, and calculates an increase amount with the passage of time of the number of times the user has viewed the target object. The specifying unit 16 specifies a product that is lost in purchase or selection based on the calculated increase amount. The output unit 17 outputs the specified lost product.

[Hardware configuration of eye-gaze analyzer]
Next, an example of the hardware configuration of the visual line analyzer 10 according to the present embodiment will be described with reference to FIG. FIG. 4 shows an example of the hardware configuration of the line-of-sight analysis apparatus 10 according to an embodiment.

  The line-of-sight analysis apparatus 10 includes an input device 101, an output device 102, an external I / F 103, a RAM (Random Access Memory) 104, a ROM (Read Only Memory) 105, a CPU (Central Processing Unit) 106, a communication I / F 107, and an HDD. (Hard Disk Drive) 108. Each part of the line-of-sight analysis apparatus 10 is connected to each other by a bus B.

  The input device 101 includes a keyboard and a mouse, and is used to input each operation signal to the line-of-sight analysis device 10. The output device 102 includes a display such as digital signage, an LCD (Liquid crystal Display) monitor, a printer, a CRT (Cathode Ray Tube), and the like, and outputs various processing results. The communication I / F 107 is an interface that connects the line-of-sight analysis apparatus 10 to a network. Thereby, the line-of-sight analysis apparatus 10 can perform data communication with a device such as a server on the cloud via the communication I / F 107.

  The HDD 108 is a non-volatile storage device that stores programs and data. The stored programs and data include basic software and application software that control the entire visual line analyzer 10. For example, the HDD 108 may store various databases, the line-of-sight analysis program 32, and the like.

  The external I / F 103 is an interface with an external device. The external device includes a recording medium 103a. Thereby, the line-of-sight analysis apparatus 10 can read and / or write to the recording medium 103a via the external I / F 103. The recording medium 103a includes a CD (Compact Disk), a DVD (Digital Versatile Disk), an SD memory card (SD Memory card), a USB memory (Universal Serial Bus memory), and the like.

  The ROM 105 is a nonvolatile semiconductor memory that can retain internal data even when the power is turned off. The ROM 105 stores programs and data such as network settings. The RAM 104 is a volatile semiconductor memory that temporarily stores programs and data. The CPU 106 is an arithmetic device that realizes control and mounting functions of the entire apparatus by reading programs and data from the storage device such as the HDD 108 and the ROM 105 onto the RAM 104 and executing processing.

  With this configuration, in the line-of-sight analysis apparatus 10 according to the present embodiment, the CPU 106 executes the line-of-sight analysis process using the line-of-sight analysis program 32 and data stored in the RAM 104, the ROM 105, and the HDD 108, for example. The information stored in the line-of-sight information DB 30 and the threshold value table 31 can be stored in the RAM 104, the HDD 108, or a server on the cloud connected to the line-of-sight analysis apparatus 10 via the network.

  FIG. 2 is a block diagram focusing on functions, and a processor that executes software of each unit indicated by these functional blocks is hardware. For example, the functions of the detection unit 12 and the acquisition unit 13 in FIG. 2 can be realized by the input device 101, for example. The function of each part of the determination part 14, the calculation part 15, and the specific | specification part 16 is implement | achieved by the process which the gaze analysis program 32 makes CPU106 perform, for example.

  The function of the output unit 17 can be realized by the output device 102, for example. The function of the storage unit 11 can be realized by the RAM 104, the ROM 105, the HDD 108, an information processing device connected to the line-of-sight analysis device 10 via a network, or other storage devices on the cloud.

<First Embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the first embodiment executed by the line-of-sight analysis apparatus 10 having the above configuration will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of a line-of-sight analysis process according to the first embodiment. The line-of-sight analysis apparatus 10 estimates the merchandise that the store visitor lost when sorting the merchandise displayed on the digital signage 20 or arranged on the display shelf.

  When this process is started, the acquisition unit 13 acquires line-of-sight data from the line-of-sight sensor 22 (step S12). Next, the determination unit 14 determines the “seen product” based on each product area from the coordinates of the acquired line-of-sight data, and stores the determination result in the “seen product” column of the line-of-sight information DB 30 (step S14). ).

  Next, the determination part 14 determines whether selection of goods was completed (step S16). Until the selection of the product is completed, the processes of step S12 and step S14 are repeated. In the present embodiment, when the selection button 24 displayed on the digital signage 20 shown in FIG. 2 is pressed, the determination unit 14 determines that the selection of the product has been completed, and proceeds to step S18.

  In step S <b> 18, the calculation unit 15 divides the time T required for product selection into three, and calculates the number of times each product is viewed in each of the three elapsed times. The time T required for product selection is the time from the start of product selection to the completion of product selection, and is counted. The following three time zones that divide time T required for product selection into three: start (product selection start time) to T / 3 elapsed time, T / 3 to 2T / 3 elapsed time, 2T / 3 to completion (product FIG. 6 shows an example of the number of times n the products A to D are viewed at each elapsed time (selection completion time). The time T required for product selection is the total time required from the start of product selection to the completion of selection by the user. For example, the unit may be seconds.

  Returning to FIG. 5, next, the calculation unit 15 calculates an increase amount of the number n of times each product is viewed (step S20). In this embodiment, the increase in the number of times n viewed for each product is 2T / 3 to completion (product selection) compared with the number of times the product was viewed within the elapsed time from start (product selection start time) to T / 3. The amount of increase in the number of times the product was viewed within the elapsed time.

  Specifically, the increment D of the number of times each product is viewed is subtracted from the number of times each product is viewed in the elapsed time of T / 3 from the number of times that each product is viewed in the elapsed time of 2T / 3 to completion. It is calculated by doing. For example, in the example of FIG. 6, the increase amount Da of the number of times the product A is viewed is −3 (= 0-3), the increase amount Db of the number of times the product B is viewed is 0 (= 1−1), The increase Dc of the number of times of viewing is calculated as 1 (= 1-0), and the increase Dd of the number of times of viewing the product D is calculated as 2 (= 2-0).

Returning to FIG. 5, next, the specifying unit 16 determines whether or not the increase D of the number of times the product has been viewed is equal to or greater than the threshold value _Sth (step S22). When it is determined that the increase D in the number of times the product has been viewed is smaller than the threshold value S _th , the specifying unit 16 determines that the target product is not a lost product (step S24), and all the products are determined. If so (step S30), the process ends. On the other hand, when it is determined that the increase in the number of times the product has been viewed is equal to or greater than the threshold value _Sth , the specifying unit 16 determines that the target product is a lost product (step S26). Referring to FIG. 1C, the last time zone obtained by dividing the time T required for product selection into three is the most recent period of the selection event, and is the time when the purchase or selection of the product is lost. Therefore, it can be estimated that the product is lost as the increase in the number of times the product in the last time zone is viewed increases with respect to the first time zone. Therefore, when it is determined that the increase D in the number of times the product has been viewed is equal to or greater than the threshold value _Sth , in the present embodiment, it is determined that the target product is a lost product. For example, it is determined that the product C and the product D that are equal to or higher than the threshold value are lost products.

  Returning to FIG. 5, the output unit 17 then outputs the “stray product” identified in step S <b> 26 (step S <b> 28). Next, the identifying unit 16 determines whether it is determined whether or not all the products are a lost product (step S30), and if it is determined whether or not all the products are a lost product, the process ends. . If it is not determined whether or not all products are a lost product, the process returns to step S22, and while there are products that are not determined whether or not they are a lost product, steps S22 to S30 are repeatedly processed.

  As described above, according to the line-of-sight analysis device 10 according to the first embodiment, the total time required for the user from the start of selection of the product to the completion of selection is, for example, T seconds, and the product is lost based on the line-of-sight data detected in T seconds. Is estimated.

  The user views the “lost object” many times at a time close to completion of selection (that is, when a selection event is detected). Therefore, in the line-of-sight analysis process according to the first embodiment, it is determined whether or not the product is a lost product based on the increase in the number of times each product has been viewed over time.

  For example, as shown in FIG. 1C, in the line-of-sight analysis process according to the first embodiment, the first elapsed time is a period in which all products are being watched in sequence, and the last product is selected. In addition, using the fact that the most recent elapsed period is a period in which some products are at a loss, a product for which selection has been lost is identified. For example, when the product D is finally selected, the products C and D are designated as “stray products” based on the increase in the number of times the last period and the first period are viewed, as shown in FIG. Can be determined.

Second Embodiment
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the second embodiment will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of a line-of-sight analysis process according to the second embodiment. The second embodiment is different from the line-of-sight analysis processing according to the first embodiment in that the threshold value serving as a determination criterion for hesitation is variable. In the line-of-sight analysis process according to the second embodiment in FIG. 7, the same step number is assigned to a step that performs the same process as the line-of-sight analysis process according to the first embodiment in FIG. 5.

  When this process is started, steps S12 to S20 are executed, and the calculation unit 15 calculates the increase in the number of times each product is viewed based on the line-of-sight data accumulated during the time T required for the product selection. Next, the calculation part 15 determines the threshold value according to the number of the goods displayed on the digital signage 20 or arrange | positioned on the display shelf (step S32).

For example, the calculation unit 15 can determine a threshold according to the number of products with reference to the threshold table 31 illustrated in FIG. When there are many choices of goods, the combination of goods for comparison increases. For this reason, since the number of times of viewing each number of products is relatively increased as compared with the case where there are few product options, the threshold value for determining the lost state of the product is also increased. In the example of FIG. 8A, S _th2 <S _th3 <S _th4 <S _th5 . The threshold value S _{th in} step S22 in FIG. 7 can be any one of S _th2 , S _th3 , S _th4 , and S _th5 .

As another example, as shown in FIG. 8 (b), the calculation unit 15 uses a threshold value corresponding to the number of products from a formula defined by S _th = an + b (a and b are coefficients set in advance). May be calculated.

Returning to FIG. 7, the specifying unit 16 determines whether the increase in the number of times the product has been viewed is equal to or greater than the determined threshold value S _th (step S _<b > 22). When it is determined that the increase in the number of times the product has been viewed is smaller than the threshold value S _th , the specifying unit 16 determines that the target product is not a lost product (step S24). On the other hand, when it is determined that the increase in the number of times the product has been viewed is equal to or greater than the threshold value _Sth , the specifying unit 16 determines that the target product is a lost product (step S26).

  When there are a lot of product choices, the number of combinations of products to be compared increases, so the number of times of viewing each product number is relatively greater than when there are few choices. Therefore, in the line-of-sight analysis apparatus 10 according to the second embodiment, a threshold for determining a lost product is variably set according to the number of products. Thereby, based on the line-of-sight data, it is possible to more accurately determine a product that the user has lost.

<Third Embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the third embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating an example of a line-of-sight analysis process according to the third embodiment.

  Depending on the user, in the time zone immediately before the selection of the product, the method of sending a line of sight that only the product that has been decided to be purchased is seen in response to the operation such as picking up the product that has been decided to purchase or pressing the selection button 24 It is possible that In this case, there is a possibility that a product that is lost but has not been purchased cannot be correctly determined as a lost product.

  For example, if the product decided to be purchased is D, the line-of-sight data of the product C that was wondering about the products C and D is not collected immediately before the selection is completed, and the product C is correctly regarded as a lost product. This is a case where it cannot be determined.

  From the above, in the third embodiment, by adding the number of times seen every elapsed time from the start of the selection of the goods in consideration of the weight of time passage, "Product" can be determined. Note that, in the line-of-sight analysis process according to the third embodiment in FIG. 9, steps that perform the same process as the line-of-sight analysis process according to the first embodiment in FIG.

  When this process is started, steps S12 to S16 are executed, and line-of-sight data is accumulated. Next, the calculation unit 15 divides the time T required for product selection into three, and calculates a value obtained by multiplying the weight corresponding to each of the three elapsed times by the number p of times each product is viewed (step S40). . For example, FIG. 10 shows a value pA obtained by multiplying a weight n (the weighting coefficients α, β, γ) corresponding to each elapsed time by the number nA of times the product A is viewed.

  The weighting factors α, β, and γ vary depending on the elapsed time. In the first embodiment, the number n of times when the product A is viewed is nA1 when the elapsed time (start to T / 3), nA2 when the elapsed time (T / 3 to 2T / 3), and the elapsed time (2T / 3). In the present embodiment, the number n actually viewed is multiplied by a weighting factor. As a result, in the third embodiment, the number of times p when the product A is viewed is pA1 (= α × nA1) at the elapsed time (start to T / 3) and at the elapsed time (T / 3 to 2T / 3). PA2 (= α × nA2) and pA3 (= α × nA3) at the elapsed time (2T / 3 to completion).

  Returning to FIG. 9, next, the calculation unit 15 calculates the weighted total values pA to pD (hereinafter, also referred to as “p total value”) of the number of times viewed for each product (step S42). For example, pA, which is the p total value of the product A, is obtained by adding a value obtained by multiplying the number of times the product is viewed at each elapsed time by a weight as shown in Expression (1).

pA = pA1 + pA2 + pA3 = α × nA1 + βnA2 + γnA3 (1)
For other products B to D, pB to pD, which are p total values of the products B to D, are calculated as shown in Equations (2) to (4).

pB = pB1 + pB2 + pB3 = α × nB1 + βnB2 + γnB3 (2)
pC = pC1 + pC2 + pC3 = α × nC1 + βnC2 + γnC3 (3)
pD = pD1 + pD2 + pD3 = α × nD1 + βnD2 + γnD3 (4)
If you are wondering about buying multiple products, the number of times you see the multiple products will increase, but only select the product you want to select immediately before completing the product selection after deciding to select or purchase one product from multiple products Turn your gaze on. In the present embodiment, considering the usage of the user's line of sight, the elapsed time weights α, β, and γ are set to values having a magnitude relationship of α ≦ β ≦ γ, so that the time elapsed weight is added. It is possible to accurately identify “a product that was rarely seen just before completion of selection but was lost”.

Next, the specifying unit 16 determines whether the p total value of the number of times viewed for each product considering the weight of elapsed time is equal to or greater than the threshold value _Zth (step S44). When it is determined that the p total value for each product is smaller than the threshold value _Zth , the specifying unit 16 determines that the target product is not a lost product (step S24). On the other hand, when it is determined that the p total value for each product is equal to or greater than the threshold value _Zth , the specifying unit 16 determines that the target product is a lost product (step S26), and outputs the lost product. (Step S28). Until all the products are determined, the processes in steps S52 and S24 to S30 are repeated. After all the products are determined, this process is terminated.

  As described above, according to the line-of-sight analysis apparatus 10 according to the third embodiment, based on the p total value for each product considering the weights α, β, and γ (α ≦ β ≦ γ) of the elapsed time, is the “lost product”? Specify whether or not. Thus, for example, when the product C or D is lost, the product selected in the product C or D but also the product that has been lost but not selected is added with the weight of time. Although it was rarely seen just before completion, it can be accurately identified as a “lost product”.

<Modification of Third Embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to a modification of the third embodiment will be described with reference to FIG. FIG. 11 is a flowchart illustrating an example of a line-of-sight analysis process according to a modification of the third embodiment. In the line-of-sight analysis processing according to the modified example of the third embodiment, the product included in the predetermined number from the top of the weighted total values pA to pD (p total value) of the number of times viewed for each product is identified as a lost product. However, it is different from the line-of-sight analysis processing according to the third embodiment, in which the product is identified as being lost as compared with the threshold value. Note that, in the line-of-sight analysis process according to the modification of the third embodiment in FIG. 11, steps that perform the same process as the line-of-sight analysis process according to the third embodiment are assigned the same step numbers.

  When this process is started, steps S12 to S16, S40, and S42 are executed, and, similarly to the third embodiment, the calculation unit 15 calculates the weighted total value pA to pD (p total value) for each product. ) Is calculated (step S42).

  Next, the calculation unit 15 uses the weighted total values pA to pD (p total value) of the number of times viewed for each product as a score, and ranks each product in descending order of score (step S50). Next, the specifying unit 16 determines whether the product is included in the predetermined order from the top in accordance with the result of ranking each product (step S52). When it is determined that the product is not within the predetermined order from the top, the specifying unit 16 determines that the target product is not a lost product (step S24). On the other hand, when it is determined that the product is within the predetermined order from the top, the specifying unit 16 determines that the target product is a lost product (step S26), and outputs the lost product (step S28). . Until all the products are determined, the processes in steps S52 and S24 to S30 are repeated. After all the products are determined, this process is terminated.

  As described above, according to the line-of-sight analysis device 10 according to the modification of the third embodiment, as in the third embodiment, the p total value for each product considering the weight of the elapsed time is used as a score, and the “lost product” is determined. Specify whether or not. Thereby, taking into account the weight of the passage of time, it is possible to accurately specify “a product that has not been seen very much immediately before completion of selection but is lost”.

<Fourth embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the fourth embodiment will be described with reference to FIG. FIG. 12 is a flowchart illustrating an example of a line-of-sight analysis process according to the fourth embodiment. In the fourth embodiment, in addition to the line-of-sight analysis processing according to the third embodiment of FIG. 9, a degree of ambiguity indicating the degree of ambiguity of the product is calculated from the P total value, and the degree of ambiguity is also calculated when the confused product is output. Is different from the third embodiment. In the line-of-sight analysis process according to the modification of the fourth embodiment in FIG. 12, steps that perform the same process as the line-of-sight analysis process according to the third embodiment in FIG.

  When this process is started, steps S12 to S16, S40, and S42 are executed, and the weighted total values pA to pD (p total values) of the number of times viewed for each product are calculated as in the third embodiment. . Next, the calculation unit 15 sets the p total value of the products having the largest weighted total value pA to pD (p total value) of the number of times viewed for each product as 1, and calculates the ratio of each product as the degree of ambiguity ( Step S54).

  Next, when the processing of steps S244 and S24 to S28 is executed, and the specifying unit 16 determines that the target product is a lost product (step S26), the lost product is displayed together with the level of doubt of the product. Output (step S28). Next, the identifying unit 16 determines whether or not all the products are lost products (step S30), and then ends the process.

  As described above, according to the line-of-sight analysis apparatus 10 according to the fourth embodiment, it is possible to output the degree of hesitation of a lost product. Thereby, it is possible to present not only the product for which the user is lost, but also how strongly the product is lost.

  In the fourth embodiment, the degree of ambiguity calculated from the P total value is replaced with the confused product in step S28 of FIG. 11 of the line-of-sight analysis process according to the modification of the third embodiment, instead of the third embodiment. It may be output.

<Fifth Embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the fifth embodiment will be described with reference to FIG. FIG. 13 is a flowchart illustrating an example of a line-of-sight analysis process according to the fifth embodiment. In the first to fourth embodiments, the time T required for selection used is divided into three, and the lost product is specified based on the number of times the product viewed for each elapsed time.

  However, if the time T required for selection used is unknown, the number of times the product has been viewed for each elapsed time corresponds to the state in which the product that is the target is being checked first, or the product selection is about to be completed. It may not be possible to distinguish whether the product corresponds to the current state, and the wrong product may be specified incorrectly.

  As shown in FIG. 14, when selecting a product, the user's line of sight is in the process of confirming each of the plurality of products and narrowing down some products from the plurality of products, and is confused soon after the product selection is completed. Transition to the state. Then, the user sees a small number of products when he / she is at a loss and finally selects a product.

  Therefore, in the line-of-sight analysis processing according to the fifth embodiment, the unit time of elapsed time is set as s, and the number n of products viewed for each elapsed time s is calculated, and the number m of products for which the viewed number n is higher than the threshold value m. Is calculated. For example, a unit time s from time (t-3) to time (t-2), a unit time s from time (t-2) to time (t-1), or more than the threshold for each unit time s. The number m of the viewed products is calculated, and the lost product is determined based on the decrease amount of m.

  Specifically, this processing is started in FIG. 13, and the acquisition unit 13 acquires line-of-sight data from the line-of-sight sensor 22 (step S12). Next, the determination unit 14 determines the product viewed from the acquired line-of-sight data, and stores the determination result in the column of the product viewed in the line-of-sight information DB 30 (step S14). Next, the acquisition unit 13 determines whether the unit time s has elapsed (step S60). The acquisition unit 13 repeats the processes of steps S12, S14, and S60 until the unit time s has elapsed.

  When it is determined that the unit time s has elapsed, the calculation unit 15 calculates the number n of times the product is viewed for each unit time s (step S62). For example, in FIG. 15, the number of times products A to I are viewed per unit time s corresponding to the passage of time (t−4), time (t−3),..., Time t, and time (t + 1). An example of the calculation result of n (nA-nI) is shown.

Returning to FIG. 13, the calculation unit 15 then counts the number m of products that satisfy the condition that the number n of products viewed in the unit time s is equal to or greater than the threshold U (for example, 3 times) of the number of products viewed per unit time s. Is calculated (step S64). The number of products m that satisfies the condition that the number of times of viewing satisfies the threshold U or more indicates the number of products that are viewed more than a predetermined number. In the following, the number of products m per unit time s corresponding to the passage of time (t-2), (t-1), t... Is m (t-2), m (t-1), m. (T) ... When the product number m (= m _old ) calculated this time is m (t−1), the product number m (= m _new ) calculated last time is m (t−2).

Next, the calculation unit 15 determines whether or not the difference between the number of products at this time and the previous time (= m _old −m _new ) is equal to or smaller than a predetermined threshold value W _{th for} m decrease (step S66). When it is determined that the difference between the current and previous product numbers is larger than the threshold value W _th , the specifying unit 16 determines that the product is not lost (step S68), and returns to the process of step S12.

In the example illustrated in FIG. 15, the selection of a product is started at time (t−4), and the number n of times each product is viewed is calculated for each unit time s. In FIG. 16A, in the example of FIG. 15, the calculation is performed for each unit time s that elapses over time (t−4), (t−3), (t−2),. The number of products m (t−4), m (t−3), m (t−2),. In this case, the decrease amount (= m _old −m _new ) of the number of products m _changes from 3 → 2 → 2 → 0 → 1 as shown in FIG.

When the threshold value W _{th for} the reduction amount of the number of products m is set to “1”, the specifying unit 16 while the reduction amount of the number of products m is determined to be larger than “1” in step S66 of FIG. Determines that he is not at a loss in selecting a product. That is, in this embodiment, as shown in FIG. 16 (b), when the reduction amount of the number of products m is larger than 1, the product selection state (the product confirmation state and product narrowing information in FIG. 14) is not lost. ).

  On the other hand, when it is determined in step S66 that the reduction amount of the number of products m is “1” or less, the specifying unit 16 determines whether the number n of times the product is viewed is “1” (step S70). . When it is determined that the number n of times the product is viewed is not “1”, the specifying unit 16 determines that the product is a lost product (step S26). The output unit 17 outputs the product determined to be lost (step S28), and ends this process.

  On the other hand, if it is determined in step S70 that the number of times the product is viewed n is “1”, there is only one product viewed in that unit time, and therefore the specifying unit 16 has completed the selection of the product. Determination is made (step S72), and this process is terminated.

  In the present embodiment, when the amount of decrease in the number of products m is 2 or more, it is determined that the product is not confused about the product selection (the product confirmation state or the product refinement information). As a result, in the example of FIG. 14, the unit time s from time (t-3) to time (t-2) and the unit time s from time (t-2) to time (t-1) are It is determined that the selection is not at a loss. On the other hand, when the reduction amount of the number of products m is smaller than 2, it is determined that the product is in a lost state or the product is completely selected. As a result. In the example of FIG. 14, the product viewed in the unit time s from time (t−1) to time t is identified as a lost product. Also, the time (t + 1) is determined as the product selection completion time.

  As described above, according to the line-of-sight analysis apparatus 10 according to the fifth embodiment, based on the line-of-sight detection status for each unit time S, the line-of-sight detection frequency or the number of detections of one unit time and the line-of-sight of the unit time immediately after It is determined whether or not the user is at a loss in selecting a product from the difference between the detection frequency or the number of detections, and the product specified as being lost is output. Thereby, even if it is a case where time T required from the start of selection of goods to completion of selection is unknown, it is possible to presume a lost state of goods near completion of selection of goods.

  As described above, according to the line-of-sight analysis device 10 according to the first to fifth embodiments, it is possible to correctly specify the target object that the user has lost selection based on the line-of-sight data. Lost objects include purchased items and unpurchased items. Thereby, it is possible to obtain information on a lost product even when the purchase is lost.

  As a result, on the side of selling the product, it is possible to promote the sale of the product based on not only the purchase status but also the non-purchase status based on the information on the lost product. In addition, on the product manufacturing side, based on the information on the lost product, it is possible to develop a product based not only on the purchase status but also on the non-purchase status.

<Sixth to eighth embodiments>
Next, after describing the functional configuration of the line-of-sight analysis apparatus 10 according to the sixth to eighth embodiments, an example of the line-of-sight analysis processing according to the sixth to eighth embodiments will be described. FIG. 17 shows an example of a functional configuration of the visual line analyzer 10 according to the sixth to eighth embodiments.

[Functional structure of eye-gaze analyzer]
Compared with the line-of-sight analysis apparatus according to the first to fifth embodiments of FIG. 2, an extraction unit 18 is added in the line-of-sight analysis apparatus 10 according to the sixth to eighth embodiments of FIG. 17. In addition, the object placement DB 33 and the lost target DB 34 are added to the storage unit 11.

  The extraction unit 18 extracts the target product that is lost from the products that have been viewed more than the threshold. FIG. 18 is a diagram illustrating an example of the object arrangement DB 33 according to an embodiment. For example, as shown in the object arrangement example on the lower side of FIG. 18, when the products A to F are arranged, the object arrangement DB 33 has the objects, that is, the arrangement of the products A to F (rows, columns). In addition, the upper left coordinates and lower right coordinates of the product are stored. FIG. 19 is a diagram illustrating an example of the lost target DB 34 according to an embodiment. In the lost target DB 34, a customer ID, a purchased product, and a lost product are stored for each customer. The lost product stores the lost product extracted by the extraction unit 18 (hereinafter, also referred to as “lost target product”). When the number of lost products is 0, it indicates that there is no candidate for the lost target product, or among the candidates for the lost target product, there is no product viewed before or after the purchase target product. When there is one lost product, it indicates that there is a product seen before or after the purchase target product among the lost target product candidates or among the lost target candidates. If there are two lost products, it indicates that there are products seen before and after the purchase target product among the candidates for the lost target product. The purchase target product is an example of a first target object.

  Note that the object arrangement DB 33 in FIG. 18 determines a product (see FIG. 3C) viewed based on each product area from the coordinates of the line-of-sight data in the line-of-sight analysis apparatus 10 according to the first to fifth embodiments. Can also be used.

<Sixth Embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the sixth embodiment will be described with reference to FIGS. FIG. 20 is a flowchart illustrating an example of a line-of-sight analysis process according to the sixth embodiment. FIG. 21 is a diagram for explaining line-of-sight analysis processing according to the sixth embodiment.

  When this process is started, the acquisition unit 13 acquires line-of-sight data from the line-of-sight sensor 22 (step S12). The acquisition method is the same as the method described in the first to fifth embodiments.

  Next, the determination unit 14 determines the “seen product” based on each product area from the coordinates of the acquired line-of-sight data, and the determination result is the “seen product” in the line-of-sight information DB 30 illustrated in FIG. The calculation unit 15 counts the number of times of viewing for each target product with the viewed product as the target product (step S80). However, instead of the number of times of viewing, the total time of viewing for each target product may be measured. The method of determining “seen product” based on each product region from the coordinates of the line-of-sight data is the same as the method described in the first embodiment.

  The number of times the target product is viewed is an example of the number of times that the target product detects the line of sight every predetermined time. The ratio of the total time viewed by the target products or the time viewed by each target product is an example of the line-of-sight detection frequency for each predetermined time of the target products.

  Note that the definition of “see” or “gazed” at a product may indicate that line-of-sight data was measured continuously for 0.4 seconds or more within the same area (radius 50 mm). However, the definition of “seen” or “watched” is not limited to this.

  Next, the extraction unit 18 defines a purchase target product (step S82). For example, when a user purchases a product displayed in a store, the product registered in the POS system may be extracted from the POS information, and the product may be a purchase target product. In addition, as for the purchase target product, the target product or the like that has been viewed most often may be defined as the purchase target product, or the purchase target product may be defined from the operation of the purchase button (vending machine, touch signage).

Next, the extraction unit 18 sets a time T _buy when the purchase target product is last viewed (step S84). In the present embodiment, as shown in FIG. 3C, T is the total time required from the start of the selection of the product to be purchased to the completion of the selection by the visitor. For example, when the last purchase item is viewed in the unit time of “20160728-134026-659”, which is the second from the bottom of FIG. 3C, the time of “20160728-134026-659” is T _buy . Show. Further, the unit time immediately before that, that is, the time of “20160728-134026-527” is indicated by (T _buy −1), and the unit time immediately after that, that is, the time zone of “20160728-134026-790” is represented by (T _buy +1).

  Returning to FIG. 20, next, the extraction unit 18 sets 1 to the variable n (step S86). The variable n takes an integer from 1 to N (N ≧ 1). N is the number of target products that are products viewed during the total time T.

  Next, the extraction part 18 determines whether the frequency | count of seeing object goods (n) is more than a threshold value (step S88). When the extraction unit 18 determines that the number of times of viewing the target product (n) is less than the threshold, the specifying unit 16 determines that the target product (n) is not a lost target product (step S90), and a variable 1 is added to n (step S98), and the process proceeds to step S100. In step S100, when the determining unit 16 determines that the variable n is equal to or less than the number N of the target products, the specifying unit 16 returns to step S88 and repeats the processing after step S88.

On the other hand, if the extraction unit 18 determines in step S88 that the number of times the target product (n) has been viewed is equal to or greater than the threshold value, the extraction unit 18 determines that the target product (n) is a candidate for the lost target product, and the target product. (N) is set as the target candidate product that has been lost (step S92). Then, the specifying unit 16, the purchase time saw target item to the end _{T buy} just before the time _(T buy -1) or purchased Last Viewed time the goods _{T buy} immediately after the time _(T buy +1) It is determined whether or not the target product (n) has been viewed (step S94).

The specifying unit 16 _applies the target product at a time immediately before the time T _{buy when} the last purchase target product is viewed (T _buy -1) or a time immediately after the time T _{buy when} the last purchase target product is viewed (T _buy +1). If it is determined that (n) is not viewed, it is determined that the target product (n) is not a lost target product (step S90). Next, the specifying unit 16 adds 1 to the variable n (step S98), and proceeds to step S100. When the variable n is determined to be equal to or less than the number N of target products, the process returns to step S88, and the processing after step S88 is performed. repeat.

In step S94, the extraction unit 18, purchased items last seen time _{T buy} just before the time of the _(T buy -1) or time _T of the time immediately after the _{buy (T buy} +1) in Shipping (n) If it is determined that the product has been viewed, the target product (n) is set to the lost product in the lost target DB 34 in FIG. 19 (step S96). Next, the extraction unit 18 adds 1 to the variable n (step S98), and proceeds to step S100. If the variable n is determined to be equal to or less than the number N of the target products, the extraction unit 18 returns to step S88, and the processing after step S88. repeat.

  In step S100, when the extraction unit 18 determines that the variable n is larger than the number N of target products, the output unit 17 outputs the lost target product as an estimation result (step S102), and ends this process. .

  As described above, in the line-of-sight analysis processing according to the sixth embodiment, based on the line-of-sight detection status for a plurality of target products, the target product with a relatively large number of detections per predetermined time for the plurality of target products is lost. As a candidate for the target product. In the present embodiment, as the target product having a relatively large number of detections per predetermined time, a product whose number of times of viewing is equal to or greater than a threshold is extracted. However, the present invention is not limited to this, and target products with a relatively large line-of-sight detection frequency or total line-of-sight time for each of the plurality of target products may be extracted as lost target product candidates.

  In the present embodiment, first, a selection event (detected from pressing of POS information or a purchase button or the like) or a selection prediction event (the number of times seen is the maximum) for a purchase target product or a purchase target prediction product described later among a plurality of target products. Detect from the target product). And at least one target among the candidates for the target product based on the detection status of the line of sight with respect to the candidate for the target product lost at least one time before or after the time when the selection event or the selection prediction event is detected Identify the product. Then, the information on the identified target product is output as a lost product.

  There is a case where a product accidentally seen just before the purchase is erroneously determined based on whether the product is viewed before or after the purchase target product. For this reason, in the present embodiment, first, a candidate with a large number of times of viewing the target product is extracted as a candidate for the target product. That is, the product viewed over the threshold is determined to be a product that the user is interested in, and is a candidate for the lost target product. As a method for extracting a candidate for a target product that has been lost, a product that has been viewed above a threshold value may be extracted, or a predetermined upper number of products that have been viewed many times may be extracted.

  Then, the lost products are narrowed down depending on whether the candidate for the lost target product is not the purchase target product and is seen immediately before or after the time when the purchase target product is selected. That is, when a candidate for a lost target product is seen at a time immediately before or immediately after a time when a purchase target product is selected, it is specified as a “lost product”.

  In other words, the user compares the “lost product” with the “purchase target product” immediately after the selection (purchase) is completed. When a purchase target product is determined from a plurality of target products, according to the above-described operation of the user, the purchase target product candidates are gradually narrowed down. The candidate for the purchase target product is also a candidate for the lost product. By extracting the lost product due to the way of comparing the last purchased target product with the lost target product, The product "can be estimated more correctly.

  For example, in the example of FIG. 21, the products B and D are extracted as candidates for the lost target product. Of these, the product B found at the time immediately before or immediately after the time when the purchase target product C is selected is set as a lost product, and the product D that is not seen at the time immediately before or immediately after the time when the purchase target product is selected. Is not set for lost products. Thereby, based on the line-of-sight data in which the user looks at the target product, it is possible to more accurately estimate the “lost product” that has not been purchased but has been purchased. Thereby, the determination accuracy of the target product that the user is interested in next to the purchase target product can be further improved.

  In the present embodiment, the user sees a candidate for the target product that is lost immediately before or immediately after the time when the purchase target product is selected (before and after the time when the purchase target product is selected is also described as (before and after ± 1)). I was judged. However, the present invention is not limited to this, and it may be determined whether or not a candidate for the target product is found at least before or after the time when the target product is selected.

  For example, it may be determined whether the user has seen a candidate for the target product that is lost within the range of the time when the purchase target product is selected (before and after ± 2), or the time when the purchase target product is selected (before and after ± 3 ), It may be determined whether a candidate for the target product is lost. Further, the time before and after the time when the purchase target product is selected may be varied according to the number N of target product viewed. For example, if the number of products is small, it is determined whether or not the candidate for the target product is lost only before and after the time when the purchase target product is selected. If the number of products is large, before and after the time when the purchase target product is selected. It may be determined whether a candidate for the target product that is lost in the range of 2 or ± 3 is seen.

Further, the threshold value used in step S88 of FIG. 20 may be changed according to the number N of the target products (the products seen) as shown in FIG. 22 (a) and FIG. 22 (b). Good. In the example of FIG. 22A, the thresholds S _{th2 to} S _th5 when the number of products is 2 to 5 are set to different values. At this time, the smaller the number of products, the higher the threshold, and S _th2 > S _th3 > S _th4 > S _th5 .

In the example of FIG. 22B, the threshold value S _th is set from the formula S _th = aT + b according to the time T required until the product is purchased. At this time, a and b are coefficients set in advance. However, the formula for _obtaining the threshold value S _th is not limited to a temporary straight line. In the example of FIG. 22, the threshold value is indicated by the number of times. However, when the total time the product is viewed is measured instead of the number of times the product is viewed, the threshold value is indicated by the time or frequency of viewing. May be.

  When n is set to a value indicating the purchase target product, the processing in steps S88 to S96 may be skipped.

<Seventh embodiment>
[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the seventh embodiment will be described with reference to FIGS. FIG. 23 is a flowchart illustrating an example of a line-of-sight analysis process according to the seventh embodiment. FIG. 24 is a diagram for explaining line-of-sight analysis processing according to the seventh embodiment.

  In the sixth embodiment, for example, the time when the product is actually purchased is set as the detection time of the selection event, and the start of the lost product determination process (steps S88 to S100 in FIG. 20).

  In contrast, in the line-of-sight analysis processing according to the seventh embodiment, when the number of line-of-sight detections per unit time of a plurality of target products becomes equal to or greater than the first determination start threshold, the product is actually purchased. Even if not, the product for which the number of line-of-sight detection is equal to or greater than the first determination start threshold is predicted as a purchase target product. The predicted purchase target product is also referred to as a purchase target prediction product. And when the selection prediction event is detected when the number of line-of-sight detections per unit time of a plurality of target products is equal to or greater than the first determination start threshold, To do. In the following description, differences from the sixth embodiment will be described.

  When this process is started, the acquisition unit 13 executes the process of step S12, and the determination unit 14 determines “seen product” based on each product region from the coordinates of the acquired line-of-sight data, and the calculation unit 15 measures the number of times of viewing for each target product (step S80).

  Next, the determination part 14 determines whether the maximum value of the frequency | count of seeing among target goods is more than a 1st determination start threshold value (step S110). FIG. 24 shows an example of the first determination start threshold. In the example of FIG. 24, the first determination start threshold has a value larger than the threshold for extracting a candidate for a lost product. If it determines with the determination part 14 not having the goods whose maximum value of the frequency | count of seeing is more than a 1st determination start threshold value among object goods, it will repeat step S80 and S110. If the determination unit 14 determines that the maximum value of the number of times of viewing is greater than or equal to the first determination start threshold, the process proceeds to step S112. In step S110, when it is determined that the maximum value of the number of times of viewing is greater than or equal to the first determination start threshold, this is an example when a selection prediction event is detected.

  In step S112, the specifying unit 16 sets a product having a maximum value of the number of times of viewing as a product most likely to be purchased (predicted product to be purchased) having the first determination start threshold value or more, and proceeds to step S84.

  Since the processes of steps S84 to S102 are the same as the processes of the steps with the same numbers in the sixth embodiment, the description thereof is omitted.

  As a result of this processing, the product C is set as the purchase target predicted product in the example of FIG. Then, among the products B and F that are candidates for the lost product, the product viewed at least one time immediately before or immediately after the time when the selection prediction event is detected is identified as the lost product.

  According to this, the product with the largest number of times of viewing is set as the purchase target prediction product. The purchase target prediction product is an example of a first target object. That is, the purchase target prediction product can be identified as a product with the highest purchase awareness in the same manner as the purchase target product, so that a product lost in real time during the measurement of the line-of-sight data can be specified. Thereby, the information regarding the specified lost product can be recommended to the user at the timing before the product purchase.

  In addition, instead of when the number of line-of-sight detections per unit time of a plurality of target products is equal to or greater than the first determination start threshold, the line-of-sight detection frequency for any unit time of the plurality of target products May be the time when the selected prediction event is detected, when the threshold value for the first determination start threshold value is exceeded or when the total line-of-sight time per unit time becomes equal to or more than the threshold value for the first determination start threshold value. .

[Gaze analysis processing]
Next, an example of a line-of-sight analysis process according to the eighth embodiment will be described with reference to FIGS. FIG. 25 is a flowchart illustrating an example of a line-of-sight analysis process according to the eighth embodiment. FIG. 26 is a diagram for explaining line-of-sight analysis processing according to the eighth embodiment.

  In the line-of-sight analysis processing according to the seventh embodiment, when the number of line-of-sight detections per unit time of a plurality of target products is equal to or greater than the first determination start threshold, the product is not actually purchased. Even if it exists, the said goods in which the frequency | count of eyes | visual_axis detection became more than the 1st determination start threshold value were set to purchase object prediction goods. And this time was set as the time of the detection of a selection prediction event, and it was set as the trigger of the start of the determination process of the lost goods.

  On the other hand, in the eighth embodiment, among a plurality of target products, the number of target products whose line-of-sight detection count per unit time is equal to or greater than a predetermined number (in this embodiment, the predetermined number is 0, that is, the number of times of viewing Is the time when the selection prediction event is detected, and is a trigger for starting the determination process of the lost product. In the following description, differences from the seventh embodiment will be described.

  When this process is started, the acquisition unit 13 executes the process of step S12, and the determination unit 14 determines “seen product” based on each product region from the coordinates of the acquired line-of-sight data, and the calculation unit 15 measures the number of times of viewing for each target product (step S80).

  Next, the determination unit 14 determines whether or not the number of target products for which the number of times of viewing has been measured among the target products is equal to or less than a second determination start threshold (step S120). FIG. 26A shows an example of the second determination start threshold.

  The determination unit 14 repeats Steps S80 and S120 while determining that the number of target products for which the number of times the target product has been viewed is measured is greater than the second determination start threshold. On the other hand, when the determination unit 14 determines that the number of target products for which the number of times the target product is viewed is measured is equal to or less than the second determination start threshold, the process proceeds to step S122. In step S120, when the number of target products for which the number of times the target product has been viewed is equal to or smaller than the second determination start threshold value, this is an example when a selection prediction event is detected. FIG. 26B shows an example of unit times T1 to T5.

  In step S122 of FIG. 25, the specifying unit 16 sets the target product with the largest number of times of viewing as a product most likely to be purchased (prediction target purchase product), and proceeds to step S84.

  Since the processing of steps S84 to S102 is the same as the processing of the step with the same number in the seventh embodiment, the description thereof is omitted.

  According to this, the item with the largest number of times of viewing is set as the purchase target prediction product. That is, the purchase target prediction product can be identified as a product with the highest purchase consciousness in the same manner as the purchase target product, thereby identifying a product lost in real time during the measurement of the line-of-sight data. Thereby, the information regarding the specified lost product can be recommended to the user at the timing before the product purchase.

  In addition, instead of when the number of target products for which the number of target products per unit time was measured is less than or equal to the second determination start threshold, the total time of the line of sight of unit time, the detection frequency, or the number of detections, The time when the number of objects equal to or less than the predetermined time, the predetermined frequency, or the predetermined number of times is equal to or less than the second determination start threshold may be set as the detection time of the selection prediction event.

  In the line-of-sight analysis processing according to the sixth to eighth embodiments and the first to fifth embodiments described above, as an event that triggers the start of line-of-sight measurement, an event such as a shelf or a signage where the line-of-sight sensor 22 is installed It may be when the user comes in front and a face (or line of sight) is detected. Moreover, it may be when it is determined that the user has stopped in front of the product shelf by detecting the position of the body or the skeleton using a camera image, a 3D sensor, or the like. Moreover, it may be when it is determined that the user has stopped in front of the product shelf by detecting the user position using Wi-Fi or GPS of the mobile terminal.

  In addition, the event that triggers the end of the line-of-sight measurement may be when the user leaves the front of the shelf or signage where the line-of-sight sensor 22 is installed and the face (or line of sight) is not detected for a predetermined time. . Alternatively, it may be determined that the user has left the product shelf by detecting the position of the body or skeleton using a camera image, a 3D sensor, or the like. Further, it may be when a lost product is specified by real-time analysis of the line-of-sight data. It may also be when the user presses a purchase button such as signage. Moreover, it may be when the measurement time of eye-gaze data set in advance has elapsed.

  The line-of-sight analysis processing according to the first to eighth embodiments can be used alone or in combination as follows. For example, in a scene where the user selects and displays products on the display shelf, by accumulating information on products that are lost compared to purchased products, based on what products the company's products are compared with, It can be used for product development such as package design.

  In addition, in a scene where the user views and selects a product displayed on the digital signage 20, when a customer touches the product to make a purchase, a recommendation (additional information, coupon information) for the product that is lost compared to the product to be purchased. ) To encourage customers to purchase.

  In both the scene where the user views and selects the products on the display shelf and the scene where the user views and selects the products displayed on the digital signage 20, the timing of analysis may be after the product purchase, You may go in real time during the previous lost time.

  As described above, the line-of-sight analysis program, the line-of-sight analysis method, and the line-of-sight analysis apparatus have been described in the above embodiment, but the line-of-sight analysis program, the line-of-sight analysis method, and the line-of-sight analysis apparatus according to the present invention are not limited to the above-described embodiment. Various modifications and improvements are possible within the scope of the invention. In addition, when there are a plurality of the above-described embodiments and modifications, they can be combined within a consistent range.

  For example, in each embodiment, the lost product is specified based on the number of times the product is viewed. However, the present invention is not limited to this. For example, the lost product may be specified based on the total time of viewing the product.

  In each embodiment, the time T required for product selection is divided into three, and the number of lines of sight for each divided elapsed time is calculated. However, the present invention is not limited to this. For example, the time T required for product selection may be divided into n, and the number of lines of sight for each elapsed time divided into n may be calculated. n may be a number of 2 or more.

  In the above embodiment, when a selection event indicating selection (purchase) of a product is detected by pressing a selection button or the like, the product is lost based on the line-of-sight detection status for a plurality of products in the time zone closest to the selection event Was identified. At this time, the most recent time zone is a time zone within a predetermined time retroactively from the detection of the selection event, or a plurality of periods T required for selection of the object from the start of the detection of the line of sight until the detection of the selection event. Of the divided periods obtained by dividing the period, a divided period relatively close to the selected event may be used.

  Further, the lost object is in a predetermined period from the start of the detection of the line of sight, or the detection frequency or the number of detections of the line of sight in the divided period that is relatively close to the detection start of the line of sight in the divided period. It may be an object in which an increase in the eye gaze detection frequency or the number of detections in the latest time zone is detected.

  Further, for example, the configuration of the line-of-sight analysis apparatus 10 according to each embodiment is merely an example, and does not limit the scope of the present invention, and various configuration examples can be made according to applications and purposes. For example, one part function of the line-of-sight analysis apparatus according to the present invention may be implemented in an information processing apparatus connected to a network or various processing apparatuses on the cloud.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
When a selection event for the first object among the plurality of objects is detected, the first object is determined based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. Identify a second object that is different from
Outputting information indicating the identified second object;
A line-of-sight analysis program that causes a computer to execute processing.
(Appendix 2)
The second object is an object that has a relatively large number of eye gaze detection frequencies or number of detections to be detected in the most recent time zone.
The line-of-sight analysis program according to supplementary note 1, wherein:
(Appendix 3)
The second object is an object having a detection frequency or number of detections of the line of sight to be detected in the most recent time zone, which is equivalent to the first object.
The line-of-sight analysis program according to Supplementary Note 1 or 2, characterized by:
(Appendix 4)
The most recent time zone is a time zone within a predetermined time retroactively from the detection of the selection event, or is obtained by dividing a period from the start of eye gaze detection to the detection of the selection event into a plurality of periods. Among the divided periods, it is a divided period that is relatively close to the selected event.
The line-of-sight analysis program according to any one of Supplementary notes 1 to 3, characterized in that:
(Appendix 5)
The second object is detected with respect to the line-of-sight detection frequency or the number of detections within a predetermined period from the start of line-of-sight detection, or within a divided period of the division period that is relatively close to the line-of-sight detection start. The object in which an increase in the detection frequency or the number of detections of the line of sight within the most recent time zone is detected.
The line-of-sight analysis program according to supplementary note 4, characterized by:
(Appendix 6)
Based on the gaze detection status of a plurality of objects for each predetermined unit time, the gaze detection frequency or the number of detections of one unit time and the gaze detection of the unit time immediately after the one unit time Determine whether or not you are confused in selecting a product from the difference in frequency or number of detections,
Gaze analysis program characterized by that.
(Appendix 7)
When a selection event for the first object among the plurality of objects is detected, the first object is determined based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. Identify a second object that is different from
Outputting information indicating the identified second object;
A line-of-sight analysis method that the computer executes.
(Appendix 8)
The second object is an object that has a relatively large number of eye gaze detection frequencies or number of detections to be detected in the most recent time zone.
The line-of-sight analysis method according to appendix 7, wherein:
(Appendix 9)
The second object is an object having a detection frequency or number of detections of the line of sight to be detected in the most recent time zone, which is equivalent to the first object.
The line-of-sight analysis method according to appendix 7 or 8, wherein
(Appendix 10)
The most recent time zone is a time zone within a predetermined time retroactively from the detection of the selection event, or is obtained by dividing a period from the start of eye gaze detection to the detection of the selection event into a plurality of periods. Among the divided periods, it is a divided period that is relatively close to the selected event.
The line-of-sight analysis method according to any one of appendices 7 to 9, characterized in that:
(Appendix 11)
The second object is detected with respect to the line-of-sight detection frequency or the number of detections within a predetermined period from the start of line-of-sight detection, or within a divided period of the division period that is relatively close to the line-of-sight detection start. The object in which an increase in the detection frequency or the number of detections of the line of sight within the most recent time zone is detected.
The line-of-sight analysis method according to appendix 10, characterized by:
(Appendix 12)
When a selection event for the first object among the plurality of objects is detected, the first object is determined based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. A specifying unit for specifying a second object different from
An output unit for outputting information indicating the identified second object;
A line-of-sight analysis apparatus.
(Appendix 13)
The second object is an object that has a relatively large number of eye gaze detection frequencies or number of detections to be detected in the most recent time zone.
Item 13. The line-of-sight analysis device according to appendix 12.
(Appendix 14)
The second object is an object having a detection frequency or number of detections of the line of sight to be detected in the most recent time zone, which is equivalent to the first object.
The line-of-sight analysis apparatus according to appendix 12 or 13, characterized in that.
(Appendix 15)
The most recent time zone is a time zone within a predetermined time retroactively from the detection of the selection event, or is obtained by dividing a period from the start of eye gaze detection to the detection of the selection event into a plurality of periods. Among the divided periods, it is a divided period that is relatively close to the selected event.
The line-of-sight analysis apparatus according to any one of appendices 12 to 14, characterized in that:
(Appendix 16)
The second object is detected with respect to the line-of-sight detection frequency or the number of detections within a predetermined period from the start of line-of-sight detection, or within a divided period of the division period that is relatively close to the line-of-sight detection start. The object in which an increase in the detection frequency or the number of detections of the line of sight within the most recent time zone is detected.
The line-of-sight analysis apparatus according to appendix 15, characterized in that.
(Appendix 17)
Based on the detection status of the line of sight for a plurality of objects, the object having a relatively high frequency of detection or the number of detections of the line of sight for each predetermined time for the plurality of objects is extracted as a candidate for the object,
When the selection event or the selection prediction event for the first object among the plurality of objects is detected, the object at the time at least either before or after the time when the selection event or the selection prediction event is detected Based on the line-of-sight detection status of the candidate, identify at least one object different from the first object from among the candidate objects,
Outputting information indicating the identified object;
A line-of-sight analysis program that causes a computer to execute processing.
(Appendix 18)
Identify at least one target object candidate based on the line-of-sight detection status of the target object candidate immediately before or immediately after the time when the selection event or selection prediction event is detected To
The line-of-sight analysis program according to supplementary note 17, characterized by:
(Appendix 19)
When any one of the detection frequency or the number of detections of the line of sight for each of the plurality of objects is equal to or higher than a first determination start threshold, a selection prediction event for the first object is detected.
The line-of-sight analysis program according to appendix 17 or 18, characterized in that.
(Appendix 20)
The selection for the first object when the detection frequency or the number of detections of the line-of-sight for each predetermined time among the plurality of objects is equal to or less than a second determination start threshold value. Detect predictive events,
The line-of-sight analysis program according to any one of supplementary notes 17 to 19, characterized in that:
(Appendix 21)
Based on the detection status of the line of sight for a plurality of objects, the object having a relatively high frequency of detection or the number of detections of the line of sight for each predetermined time for the plurality of objects is extracted as a candidate for the object,
When the selection event or the selection prediction event for the first object among the plurality of objects is detected, the line-of-sight detection of the candidate for the object at the time before and after the time when the selection event or the selection prediction event is detected. Based on the situation, identify at least one object different from the first object from among the candidate objects,
Outputting information indicating the identified object;
A line-of-sight analysis method that the computer executes.
(Appendix 22)
Identify at least one target object candidate based on the line-of-sight detection status of the target object candidate immediately before or immediately after the time when the selection event or selection prediction event is detected To
Item 22. The line-of-sight analysis method according to appendix 21.
(Appendix 23)
When any one of the detection frequency or the number of detections of the line of sight for each of the plurality of objects is equal to or higher than a first determination start threshold, a selection prediction event for the first object is detected.
23. The line-of-sight analysis method according to appendix 21 or 22,
(Appendix 24)
The selection for the first object when the detection frequency or the number of detections of the line-of-sight for each predetermined time among the plurality of objects is equal to or less than a second determination start threshold value. Detect predictive events,
The line-of-sight analysis method according to any one of appendices 21 to 23, characterized in that:
(Appendix 25)
An extraction unit that extracts, as candidates for objects, objects having a relatively high line-of-sight detection frequency or number of detections for each predetermined time based on the line-of-sight detection status of the plurality of objects. When,
When the selection event or the selection prediction event for the first object among the plurality of objects is detected, the line-of-sight detection of the candidate for the object at the time before and after the time when the selection event or the selection prediction event is detected. A specifying unit for specifying at least one object different from the first object from among the candidates for the object based on a situation;
An output unit that outputs information indicating the identified object;
A line-of-sight analysis apparatus.
(Appendix 26)
The specifying unit is configured to select at least one of the object candidates based on a gaze detection state of the object candidate at least immediately before or immediately after the time when the selection event or the selection prediction event is detected. Identify one feature,
The line-of-sight analysis apparatus according to appendix 25, wherein:
(Appendix 27)
The extraction unit is configured to perform selection prediction for the first object when any of the line-of-sight detection frequency or the number of detections for the plurality of objects reaches or exceeds a first determination start threshold value. Detect events,
27. The line-of-sight analysis apparatus according to appendix 25 or 26, wherein
(Appendix 28)
The selection for the first object when the detection frequency or the number of detections of the line-of-sight for each predetermined time among the plurality of objects is equal to or less than a second determination start threshold value. Detect predictive events,
The line-of-sight analysis apparatus according to any one of Supplementary Notes 25 to 27, characterized in that:

DESCRIPTION OF SYMBOLS 10 Eye-gaze analyzer 11 Memory | storage part 12 Detection part 13 Acquisition part 14 Determination part 15 Calculation part 16 Specification part 17 Output part 18 Extraction part 30 Eye-gaze information DB
31 Threshold Table 32 Gaze Analysis Program 33 Object Placement DB
34 Target DB lost

Claims (14)

When a selection event for the first object among the plurality of objects is detected, the first object is determined based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. Identify a second object that is different from
Outputting information indicating the identified second object;
A line-of-sight analysis program that causes a computer to execute processing. The second object is an object that has a relatively large number of eye gaze detection frequencies or number of detections to be detected in the most recent time zone.
The line-of-sight analysis program according to claim 1. The second object is an object having a detection frequency or number of detections of the line of sight to be detected in the most recent time zone, which is equivalent to the first object.
The line-of-sight analysis program according to claim 1 or 2. The most recent time zone is a time zone within a predetermined time retroactively from the detection of the selection event, or is obtained by dividing a period from the start of eye gaze detection to the detection of the selection event into a plurality of periods. Among the divided periods, it is a divided period that is relatively close to the selected event.
The line-of-sight analysis program according to any one of claims 1 to 3. The second object is detected with respect to the line-of-sight detection frequency or the number of detections within a predetermined period from the start of line-of-sight detection, or within a divided period of the division period that is relatively close to the line-of-sight detection start. The object in which an increase in the detection frequency or the number of detections of the line of sight within the most recent time zone is detected.
The line-of-sight analysis program according to claim 4. Based on the gaze detection status of a plurality of objects for each predetermined unit time, the gaze detection frequency or the number of detections of one unit time and the gaze detection of the unit time immediately after the one unit time Determine whether or not you are confused in selecting a product from the difference in frequency or number of detections,
Gaze analysis program characterized by that. When a selection event for the first object among the plurality of objects is detected, the first object is determined based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. Identify a second object that is different from
Outputting information indicating the identified second object;
A line-of-sight analysis method that the computer executes. When a selection event for the first object among the plurality of objects is detected, the first object is determined based on the line-of-sight detection status of the plurality of objects in the time zone closest to the selection event. A specifying unit for specifying a second object different from
An output unit for outputting information indicating the identified second object;
A line-of-sight analysis apparatus. Based on the detection status of the line of sight for a plurality of objects, the object having a relatively high frequency of detection or the number of detections of the line of sight for each predetermined time for the plurality of objects is extracted as a candidate for the object,
When the selection event or the selection prediction event for the first object among the plurality of objects is detected, the object at the time at least either before or after the time when the selection event or the selection prediction event is detected Based on the line-of-sight detection status of the candidate, identify at least one object different from the first object from among the candidate objects,
Outputting information indicating the identified object;
A line-of-sight analysis program that causes a computer to execute processing. Identify at least one target object candidate based on the line-of-sight detection status of the target object candidate immediately before or immediately after the time when the selection event or selection prediction event is detected To
The line-of-sight analysis program according to claim 9. When any one of the detection frequency or the number of detections of the line of sight for each of the plurality of objects is equal to or higher than a first determination start threshold, a selection prediction event for the first object is detected.
The line-of-sight analysis program according to claim 9 or 10. The selection for the first object when the detection frequency or the number of detections of the line-of-sight for each predetermined time among the plurality of objects is equal to or less than a second determination start threshold value. Detect predictive events,
The line-of-sight analysis program according to any one of claims 9 to 11. Based on the detection status of the line of sight for a plurality of objects, the object having a relatively high frequency of detection or the number of detections of the line of sight for each predetermined time for the plurality of objects is extracted as a candidate for the object,
When the selection event or the selection prediction event for the first object among the plurality of objects is detected, the line-of-sight detection of the candidate for the object at the time before and after the time when the selection event or the selection prediction event is detected. Based on the situation, identify at least one object different from the first object from among the candidate objects,
Outputting information indicating the identified object;
A line-of-sight analysis method that the computer executes. An extraction unit that extracts, as candidates for objects, objects having a relatively high line-of-sight detection frequency or number of detections for each predetermined time based on the line-of-sight detection status of the plurality of objects. When,
When the selection event or the selection prediction event for the first object among the plurality of objects is detected, the line-of-sight detection of the candidate for the object at the time before and after the time when the selection event or the selection prediction event is detected. A specifying unit for specifying at least one object different from the first object from among the candidates for the object based on a situation;
An output unit that outputs information indicating the identified object;
A line-of-sight analysis apparatus.

Images