JP7206806B2 - Information processing device, analysis method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an analysis method, and a program.

In recent years, the provision of various services using AR (Augmented Reality) has been considered. For example, in a store where products are bought and sold, when a customer visually recognizes the product through the display unit of a smartphone or tablet, we provide a service that displays the image of the product and the description of the product on the display unit. is being considered. Japanese Patent Laid-Open No. 2004-100002 discloses a process of displaying additional information of a product displayed in a video image captured using a mobile terminal such as a smartphone.

WO2014/017392

In the future, in order for stores to efficiently increase sales, it is desired to analyze the purchasing behavior of users who purchase products using information terminals and to operate stores using the analysis results. Patent Literature 1 discloses the use of AR to encourage the purchase of a product using an information terminal. However, in the system disclosed in Patent Document 1, there is a problem that it is impossible to acquire information on customer's purchasing behavior from the information terminal and to analyze the acquired information.

An object of the present disclosure is to provide an information processing device, an analysis method, and a program that can acquire information from an information terminal that utilizes AR and analyze a user's purchasing behavior using the acquired information.

An information processing apparatus according to a first aspect of the present disclosure includes an image acquisition unit that acquires a captured image in which an object is captured, and a display that superimposes information related to the object on the captured image and displays it on a display unit. a control unit, acquiring positions of the target object and the related information within the display unit, and using a predetermined position within the display unit and positions of the target object and the related information to display the display unit; an analysis unit that analyzes interest of a user viewing the object through the object.

An analysis method according to a second aspect of the present disclosure acquires a captured image in which an object is captured, superimposes information related to the object on the captured image and displays it on a display unit, and displays the Obtaining a position of an object and the related information, and using a predetermined position within the display unit and the position of the object and the related information within the display unit, the object through the display unit An analysis method for analyzing the interests of users who view the .

A program according to a third aspect of the present disclosure acquires a captured image in which an object is captured, superimposes relevant information of the object on the captured image and displays it on a display unit, and displays the object in the display unit. obtaining positions of an object and the related information, and displaying the object through the display unit using a predetermined position within the display unit and the positions of the object and the related information within the display unit; A computer is made to analyze the viewing user's interest.

According to the present disclosure, it is possible to provide an information processing device, an analysis method, and a program capable of acquiring information from an information terminal that utilizes AR and analyzing user purchasing behavior using the acquired information.

1 is a configuration diagram of an information processing apparatus according to a first embodiment; FIG. 1 is a configuration diagram of an information processing apparatus according to a second embodiment; FIG. FIG. 10 is an image diagram of an image displayed on the display unit according to the second embodiment; FIG. FIG. 10 is an image diagram of an image displayed on the display unit according to the second embodiment; FIG. FIG. 10 is an image diagram of an image displayed on the display unit according to the second embodiment; FIG. FIG. 11 is a diagram for explaining analysis processing according to the second embodiment; FIG. FIG. 10 is an image diagram of an image displayed on the display unit according to the second embodiment; FIG. FIG. 10 is an image diagram of an image displayed on the display unit according to the second embodiment; FIG. FIG. 10 is a diagram showing the flow of analysis processing in the information processing apparatus according to the second embodiment; FIG. 11 is a diagram for explaining analysis processing according to a modification of the second embodiment; FIG. FIG. 11 is a diagram for explaining analysis processing according to a modification of the second embodiment; FIG. 1 is a configuration diagram of an information processing apparatus according to each embodiment; FIG.

(Embodiment 1)
Embodiments of the present disclosure will be described below with reference to the drawings. A configuration example of the information processing apparatus 10 will be described with reference to FIG. The information processing device 10 may be a computer device operated by a processor executing a program stored in a memory. The information processing device 10 may be, for example, an information terminal such as a smartphone terminal and a tablet terminal. Alternatively, the information processing device 10 may be a personal computer device, a server device, or the like.

The information processing device 10 has an image acquisition unit 11 , a display control unit 12 and an analysis unit 13 . Components constituting the information processing apparatus 10 such as the image acquisition unit 11, the display control unit 12, and the analysis unit 13 are software or modules whose processing is executed by the processor executing a program stored in the memory. may Alternatively, the components of the information processing device 10 may be hardware such as circuits or chips.

The image acquisition unit 11 acquires a captured image in which an object is captured. The object may be, for example, a product displayed in a store. Specifically, the object may be a product that the customer has purchased or is considering purchasing. Alternatively, the object may be a device operated by the user or a component of the device. Specifically, the object may be a device to be maintained by an administrator or the like.

The captured image may be a still image or a moving image. The image acquisition unit 11 may acquire a captured image captured using a camera built into the information processing apparatus 10, or may acquire a captured image captured using a camera externally attached to the information processing apparatus 10. may be obtained. Alternatively, the image acquisition unit 11 may acquire a captured image captured by a remotely installed camera via a network such as the Internet or an intranet. Alternatively, the image acquisition unit 11 may acquire the captured image from a camera connected via a wireless communication line. As the wireless communication line, for example, wireless LAN (Local Area Network) communication may be used, or short-range wireless communication such as Bluetooth (registered trademark) may be used. A user may use the information processing apparatus 10 having an imaging function of an object as an information terminal, or may use a camera having an imaging function and a communication function of transmitting a captured image to another apparatus as an information terminal.

The display control unit 12 superimposes the information related to the object on the captured image and causes the display unit to display the information. The related information may be information indicating, for example, the name of the object, the characteristics of the object, the components contained in the object, the place of production or manufacturing of the object, and the like. Furthermore, when the object is food, the related information may be information indicating how to cook the object. Alternatively, the related information may be information that prompts the customer to purchase the target object. The information prompting the purchase of the target object may be, for example, discount information of the target object, information indicating the popularity of the target object, or the like. Alternatively, the related information may be information indicating an operation manual for the device, functions of buttons included in the device, and the like. The related information may be text data, still image data, or moving image data. The user may select the language in which the related information will be displayed. For example, the user may enter information regarding the selected language. Specifically, the user may select a language by performing a touch operation or a button operation on the display unit. The related information may be information displayed on the display using AR, and may be referred to as AR information.

To superimpose the relevant information of the object on the captured image means, for example, to generate video data or image data so that the relevant information is displayed in the vicinity of the object or in a position overlapping with a part of the object. In other words, superimposing the related information of the object on the captured image may be combining the captured image and the related information, which is image data.

The display control unit 12 may extract related information associated with the object from a memory in the information processing device 10 or a memory externally attached to the information processing device 10, or extract related information from another device via a network. Related information may be received. For example, the display control unit 12 may identify an object included in the captured image and extract related information associated with the identification information of the object. Alternatively, the information processing device 10 may transmit the identification information of the object to another device via the network and receive related information associated with the identification information from the other device.

The display unit may be, for example, a display device. The display unit may be configured integrally with the information processing device 10, may be connected to the information processing device 10 via a cable or wireless communication, or may be connected to the information processing device 10 via a network. . Displaying by the display control unit 12 on the display unit may be translated into outputting the image data generated by the display control unit 12 to the display unit.

The analysis unit 13 acquires the positions of the target object and related information within the display unit. Further, the analysis unit 13 analyzes the interest of the user viewing the object through the display unit using the predetermined position within the display unit and the positions of the object and related information.

The position of the object and related information in the display unit is, for example, the position on the display of the object and related information displayed on the display. The position of the object and related information may be any position within the area where the object and related information are displayed. For example, the position of the target object and related information may be a position indicating the center of the target object and related information, or an arbitrary position on the contour of the target object and related information. The position may be indicated using coordinates on the XY plane when the surface of the display is the XY plane. The analysis unit 13 may acquire the positions of the target object and the related information from the display control unit 12 that generates the image data, and acquire the positions of the target object and the related information from the display unit that displays the image data. good too.

The predetermined position within the display unit may be, for example, an arbitrary position on the display predetermined as a position where the user's line of sight is most concentrated. The predetermined position within the display may be, for example, the center on the display. A predetermined position within the display unit may be changed by, for example, a user's operation.

The user's interest may be, for example, what kind of information the user focused on to purchase the target object, the user's preference, or the like. Alternatively, the user's interest may be, for example, which part of the device, such as buttons and gauges, the user pays attention to when operating the device according to the operation manual indicated in the related information. The user's interest may be information indicating the user's behavior or information that may be the basis for the user's behavior.

Using the predetermined position in the display section and the position of the object and related information means defining the relationship between the predetermined position in the display section and the position of the target object and related information. may For example, the distance from a predetermined position within the display unit to the position of the object and related information may be calculated. For example, the analysis unit 13 may analyze that the shorter the calculated distance, the higher the interest in the object or related information.

As described above, the information processing apparatus 10 can acquire the positions of the target object and related information and the predetermined positions within the display unit. The position of the object and related information relative to the predetermined position within the display is information that changes according to the user's interest in or interest in the object or related information. Therefore, the information processing apparatus 10 can analyze the user's interest in the target object or related information by acquiring the position of the target object and related information and the predetermined position within the display unit. In other words, the information processing device 10 can analyze the behavior of the user.

(Embodiment 2)
Next, a configuration example of the information processing apparatus 20 according to the second embodiment will be described with reference to FIG. The information processing device 20 has a configuration in which an imaging unit 21, a display unit 22, and a related information generation unit 23 are added to the configuration of the information processing device 10 in FIG. In the information processing device 20, the detailed description of the configuration similar to that of the information processing device 10 is omitted. The information processing device 20 may be, for example, a smartphone terminal or a tablet terminal.

The imaging unit 21 images an object and generates a captured image. The imaging unit 21 outputs the captured image to the image acquisition unit 11 as video data, which is digital data. The object to be imaged changes according to the operation of the user who operates the information processing device 20 . For example, the user captures an image of the object by holding the smartphone terminal, which is the information processing device 20, in front of the object.

The image acquisition unit 11 outputs video data acquired from the imaging unit 21 to the display control unit 12 . Furthermore, the image acquisition unit 11 performs image recognition processing using video data. The image acquisition unit 11 identifies objects included in the video data by performing image recognition processing. Image recognition processing for identifying an object is performed, for example, by extracting a feature amount in video data, and comparing the extracted feature amount with the feature amount of the object registered in advance in the memory or the like in the information processing apparatus 20. match. If the extracted feature amount matches the feature amount of the registered target object or if the number of matching portions exceeds a predetermined value, the image acquisition unit 11 detects the target object registered in advance in the video data. identify it as containing The image acquisition unit 11 outputs identification information of the specified object to the related information generation unit 23 .

The related information generator 23 associates and manages the identification information of the object and the related information of the object. For example, if the target object is a product displayed in a store, the product name may be used as the product identification information. Furthermore, the name of the product, the component information of the product, the characteristic information of the product, the production area information of the product, and the like may be associated and managed. When acquiring the identification information from the image acquisition unit 11 , the related information generation unit 23 outputs related information associated with the identification information to the display control unit 12 . When the user inputs selection information that selects related information to be displayed to the information processing apparatus 10, the related information generation unit 23 displays the selected related information among the related information associated with the identification information. Output to the control unit 12 . If the user has not input the selection information to the information processing device 10, the related information generation unit 23 may output any related information among the related information associated with the identification information to the display control unit 12. . Alternatively, the related information generator 23 may sequentially output a plurality of related information associated with the identification information to the display controller 12 at predetermined intervals. Alternatively, the related information generation unit 23 may preset a priority order for each of a plurality of related information, and output the related information to the display control unit 12 according to the priority order.

The display control unit 12 generates display data in which the related information output from the related information generation unit 23 is superimposed on the video data output from the image acquisition unit 11 . The display control unit 12 may superimpose the related information at a predetermined position in the video data, for example. Alternatively, the display control unit 12 may specify the position of the object included in the video data and superimpose the related information on the video data so as not to overlap with the object. Alternatively, the display control unit 12 may specify the position of the object included in the video data and superimpose the related information on the video data so as to partially overlap the object. The display control unit 12 outputs the generated display data to the display unit 22 and the analysis unit 13 .

The display unit 22 displays display data output from the display control unit 12 . That is, the display data output from the display control unit 12 is displayed on the display. The user visually recognizes the target object and related information through the image displayed on the display unit 22 . Also, here, a point or area on the display where the user's line of sight is most concentrated is referred to as a gaze point. In other words, the gaze point indicates where the user pays the most attention on the display on which the object and related information are displayed. The gaze point may be, for example, the center on the display, or may be another location.

While looking at the image displayed on the display, the user aligns the point of interest with the point of interest in the image. In other words, the user changes the image displayed in a predetermined area (hereinafter referred to as a gaze area) including the gaze point. For example, the user adjusts the angle of the imaging unit 21 with respect to the object to change the image displayed in the gaze area. For example, if the user is more interested in the object than in the related information, the object is displayed in the gaze area. Alternatively, if the user is more interested in the related information than the object, the related information is displayed in the gaze area. The imaging unit 21 outputs, for example, an image while the user is changing the image displayed in the gaze region to the image acquisition unit 11 as moving image data. The display control unit 12 outputs display data generated by superimposing the related information acquired from the related information generation unit 23 on the moving image data acquired from the image acquisition unit 11 to the display unit 22 and the analysis unit 13 .

From the display data acquired from the display control unit 12, the analysis unit 13 identifies the position of the gaze point in the image and the positions of the target object and related information. Further, after the target object and related information are displayed on the display unit 22, the user changes the positions of the target object and the related object displayed on the display unit 22 according to his or her interest. Therefore, the analysis unit 13 acquires the position of the point of gaze in the image and the positions of the object and related information from the display control unit 12 at predetermined intervals, The location of objects and related information may be used to analyze user interests. Alternatively, the analysis unit 13 determines the position of the point of gaze after the amount of change in the distance from the position of the point of gaze to the position of the object and related information in a predetermined period of time becomes smaller than a predetermined value, The location of relevant information may be used to analyze user interests. Here, the image of the image displayed on the display unit 22 and the positions of the gaze point, the target object, and the related object will be described with reference to FIG. 3 .

In FIG. 3, the display area 30 shows a PET bottle as an object. The display area 30 is an area on the display where an image is displayed. Also, the gaze point 31 is defined at the center of the display area 30 . The gaze area 35 may be an area within a circle centering on the gaze point 31 . Alternatively, the gaze area 35 may be a closed area formed using curved lines, straight lines, or curved lines and straight lines, and may be an area including the gaze point 31 . Furthermore, a product label 32 and an ingredient label 33 are shown on the PET bottle. Further, the display area 30 shows related information 34 of the PET bottle. FIG. 3 shows an example in which the product label 32 and the ingredient label 33 are displayed in the display area 30, but the positions of the product label 32 and the ingredient label 33 are not specified as shown in FIG. The display control unit 12 generates display data to be displayed in the display area 30 .

FIG. 3 shows that the point of gaze is on the product label 32 on the PET bottle. That is, the product label 32 is displayed in the gaze area 35 . Displaying the product label 32 in the gaze area 35 may mean that only the product label 32 is displayed in the gaze area 35 or that part of the product label 32 is displayed in the gaze area 35 . may In this case, it is presumed that the user operating the information processing device 10 is paying attention to the product label 32 .

The analysis unit 13 may calculate the distance between the point of interest 31 and the product label 32, the distance between the point of interest 31 and the component label 33, and the distance between the point of interest 31 and the related information 34, for example. In the example of FIG. 3, since the point of gaze 31 exists on the product label 32, the distance between the point of gaze 31 and the product label 32 may be calculated as zero. The distance between the point of interest 31 and the component label 33 and the distance between the point of interest 31 and the related information 34 may be calculated using coordinates on the XY plane, for example. FIG. 3 shows that the position of the gaze point 31 and the position of the product label 32 overlap, and the product label 32 is displayed in the gaze point area. Further, the analysis unit 13 may calculate the distance between the gaze point 31 and the area center of gravity position of the product label 32 . Alternatively, the area centroid position may be the centroid position of the entire product and the related information 34 . Further, the related information 34 may be analyzed by analyzing the user's interest according to the information displayed in the gaze area 35 with the center of gravity of the entire product and the related information 34 as the gaze point 31 .

Furthermore, the analysis unit 13 may count the period during which the product label 32 is displayed in the gaze area 35 . In other words, the analysis unit 13 may count periods in which the point of gaze 31 and the product label 32 overlap. Furthermore, the analysis unit 13 may specify the timing when the product label 32 is displayed in the gaze area 35 and the timing when the product label 32 is no longer displayed in the gaze area 35 . For example, the analysis unit 13 may specify the time when the product label 32 is displayed in the gaze area 35 and the time when the product label 32 is no longer displayed in the gaze area 35 . Alternatively, the analysis unit 13 may identify the timing when the point of gaze 31 and the product label 32 overlap and the timing when the point of gaze 31 and the product label 32 no longer overlap.

FIG. 4 shows that the point of regard 31 is at the component label 33 . In this case, it is presumed that the user operating the information processing device 10 is paying attention to the component label 33 . FIG. 5 shows that the gaze point 31 exists in the related information 34 . In this case, it is presumed that the user operating the information processing apparatus 10 is paying attention to the related information 34 .

Next, an example of analysis processing by the analysis unit 13 will be described with reference to FIG. Arrows in FIG. 6 indicate the passage of time. Let time t1 be the time when the user comes to the store.

After that, the analysis unit 13 uses the display data output from the display control unit 12 to identify that the product label of the product A was displayed in the gaze area from time t2 to time t3. Next, the analysis unit 13 identifies that related information was displayed in the gaze area from time t3 to time t4. Next, the analysis unit 13 identifies that the ingredient label of product A was displayed in the gaze area from time t4 to time t5. Next, the analysis unit 13 identifies that the component label of product B was displayed in the gaze area from time t5 to time t6. Next, the analysis unit 13 identifies that the component label of product A was displayed in the gaze area from time t6 to time t7.

FIG. 6 shows that the user paid more attention to product A than product B for a longer period of time. Therefore, the analysis unit 13 may analyze that the user is more interested in product A than product B. Also, FIG. 6 shows that the user paid more attention to product A than to product B. In FIG. Therefore, the analysis unit 13 may analyze that the user is more interested in product A than product B. Furthermore, FIG. 6 shows that the user paid more attention to the ingredient labels than the product labels of products A and B for a longer period of time. Therefore, the analysis unit 13 may analyze that the component of the product is of more interest than the name of the product. Also, FIG. 6 shows that the user pays more attention to the product label or ingredient label than to the related information for a longer period of time. Therefore, the analysis unit 13 may analyze that when purchasing a product, the consumer is more interested in the product label or ingredient label displayed on the product itself than in the related information.

Further, the analysis unit 13 may analyze the user's interest in a plurality of pieces of information related to one product using the order of the information that the user paid attention to. For example, FIG. 6 shows that the user focused on the product label for product A and then on the ingredient label for product A. FIG. In such a case, the analysis unit 13 may analyze that it is more interested in the ingredient label than the product label. Alternatively, the analysis unit 13 may analyze that the user is more interested in the product label of the product A that the user first paid attention to than the ingredient label.

In addition, the analysis unit 13 may analyze the user's interest in a plurality of pieces of information written in the component label using the order of the information that the user paid attention to. For example, FIG. 7 shows that in display area 30 , point of regard 31 resides in a stored method within component label 33 . Next, FIG. 8 shows that the point of regard 31 has moved to the ingredients in the ingredient label. In other words, FIGS. 7 and 8 show that after the user has focused on the preservation method in the ingredient label 33, he has focused on the ingredients. In such a case, the analysis unit 13 may analyze that the raw material is of more interest than the storage method. Alternatively, the analysis unit 13 may analyze that the user is more interested in the storage method that he first noticed than in the raw material. The analysis unit 13 may analyze the user's interest in a plurality of pieces of information described in the product label using the order of the pieces of information that the user paid attention to.

The analysis unit 13 may analyze the user's interest based on the content displayed as AR information in the related information. For example, when the features of a specific component are displayed in detail using text as AR information, and the gaze point 31 exists in the AR information, it may be analyzed that there is an interest in the component of the product. Also, if a commercial or promotional video of a product is displayed as AR information and the gaze point 31 exists in the AR information, it may be analyzed that the user is interested in the brand or image of the product. Also, when word-of-mouth information about a product is displayed as AR information and the gaze point 31 exists in the AR information, it may be analyzed that the user is interested in the popularity of the product or other people's opinions. Also, if a discount rate or the like is displayed as AR information and the gaze point 31 exists in the AR information, it may be analyzed that the user is interested in the price of the product.

Here, it is assumed that the analysis unit 13 acquires information indicating that the user has purchased the product A. For example, the information processing device 20 may be connected to a network such as an intranet built inside the store. Specifically, the information processing device 20 may be connected to an access point, which is a master device arranged on the network, as a slave device of a wireless LAN (Local Area Network). In this case, the analysis unit 13 may acquire information indicating that the user purchased the product A from the POS (Point Of Sales) system of the store connected to the intranet. Alternatively, the information processing device 20 may perform short-range wireless communication with the basket storing the product, and determine that the user has purchased the product A when notified that the product A is stored in the basket. . For example, the reader device attached to the basket may detect that the product A is stored in the basket by reading the tag information of the product A. Further, the information processing device 20 may receive information indicating that the product A has been stored in the car through Bluetooth (registered trademark) communication between the information processing device 20 and the car. Alternatively, the user may input information on the purchased product to the information processing device 20 .

When the analysis unit 13 acquires information indicating that the user has purchased product A, the analysis unit 13 determines that the component information of product A that has been displayed in the gaze region for the longest time is the reason why the user has purchased product A. may be analyzed.

Furthermore, the analysis unit 13 may estimate the attributes of the user based on the analysis processing in FIG. 6 or information such as the user's purchased products. For example, the analysis unit 13 may estimate that the user is male when an item that has been displayed in the gaze area for longer than a predetermined threshold is likely to be purchased by men. . Moreover, the analysis unit 13 may estimate that the user is a woman when the product purchased by the user is likely to be purchased by a woman. Furthermore, the analysis unit 13 may estimate the user's age based on the product that the user pays attention to or the purchased product. Furthermore, the analysis unit 13 may estimate the user's residential area based on the language selected by the user as the language displayed in the related information.

The analysis unit 13 classifies items estimated regarding the purchasing behavior of each user into specific attributes and generates marketing information. The marketing information generated by the analysis unit 13 may be transmitted to, for example, a server device or the like installed inside the store, or may be transmitted to a server device or the like located outside the store via the Internet. good too.

Furthermore, the analysis unit 13 may analyze the products purchased by the user and further specify products to be proposed to the user for purchase. The analysis unit 13 may use Bayesian inference to identify products to be purchased by the user. Alternatively, the analysis unit 13 may manage a database that defines a product group including a plurality of related products in advance. In this case, the analysis unit 13 may specify other products in the product group to which the product purchased by the user belongs as products to be proposed for purchase. A product group may be, for example, a plurality of products having the same ingredients, or a plurality of products used in the same event. The analysis unit 13 may output information about the product identified as the product to be purchased to the display control unit 12 , and the display control unit 12 may cause the display unit 22 to display the product.

Furthermore, the analysis unit 13 may further analyze the information of interest in the purchase of the product, and specify the product to be proposed to the user for purchase. For example, it is assumed that the analysis unit 13 analyzes that the user is interested in the ingredient label when purchasing a product in the analysis process using FIG. 6 . Furthermore, it is assumed that the analysis unit 13 acquires information indicating that the user has purchased product A. In this case, the analysis unit 13 may further specify other products containing the ingredients included in the ingredient label of product A as products to be proposed for purchase. The product to be purchased may be in the same category as the product A, or may be in a category different from the product A.

Alternatively, it is assumed that the analysis unit 13 has analyzed in the analysis process using FIG. 6 that the user is interested in the ingredient label when purchasing the product. In this case, other products that contain ingredients commonly included in the ingredient label of product A and the ingredient label of product B may be specified as products to be purchased. In addition, the product suggested for purchase may be a product displayed in the same store as the store where the user is shopping, or may be a product existing in a nearby store.

Furthermore, the analysis unit 13 may output to the related information generation unit 23 information about the product specified as the product for which purchase is proposed. In this case, the related information generation unit 23 may generate related information indicating a flow line to the place where the product is displayed in the store. In this case, it is assumed that the related information generation unit 23 manages in advance map information in the store and information on the display location of the product.

Next, the flow of analysis processing in the information processing device 20 will be described with reference to FIG. 9 . First, the image acquiring unit 11 acquires video data, which is a captured image, from the imaging unit 21, and identifies products included in the video data (S11). The image acquisition unit 11 may acquire video data including multiple products. Alternatively, the image acquisition unit 11 may acquire the image data including the product B after acquiring the image data including the product A. In other words, the image acquisition unit 11 may acquire video data in which a plurality of commodities are imaged at the same timing, or may acquire video data relating to a plurality of commodities imaged at different timings.

Next, the display control unit 12 superimposes the product related information output from the related information generation unit 23 onto the video data to generate display data (S12). Next, the analysis unit 13 acquires point-of-regard information (S13). The point-of-regard information may be information indicating the position of the point-of-regard, the position of the product, and the position of the related information. The point-of-regard information may also include the distance between the point-of-regard and the product, and the distance between the point-of-regard and the related information. The distance between the point of gaze and the product, and the distance between the point of gaze and the related information may be calculated by the analysis unit 13 .

Further, the point-of-regard information may include information regarding the period during which the point-of-regard overlaps with the product or related information. Furthermore, the point-of-regard information may include information indicating the timing at which the point-of-regard and the product or related information overlap and the timing at which the point-of-regard and the product or related information no longer overlap.

Next, the analysis unit 13 analyzes the user's interest using the point-of-regard information (S14). For example, the analysis unit 13 executes analysis processing shown in FIG.

Next, the analysis unit 13 estimates the attributes of the user based on the result of the analysis process executed in step S14 or information such as the product purchased by the user (S15). For example, the analysis unit 13 may estimate the user's sex, age, place of residence, etc. as attributes.

Next, the analysis unit 13 classifies the items estimated regarding the product purchasing behavior of each user into specific attributes, and generates marketing information (S16).

As described above, the information processing apparatus 20 can analyze the user's interest based on the positional relationship between the gaze point and the target object and related information. Specifically, the information processing device 20 can analyze what kind of information the user is interested in and what kind of information the user has decided to purchase the product based on.

In addition, the information processing device 20 may analyze the behavior of workers during maintenance work on a production line or server. For example, when a work procedure or operation manual is displayed as AR information, which is related information, the work of the worker is analyzed using the position where the worker is working, the position of the line of sight, the work time, etc. may

(Modification of Embodiment 2)
Next, an example of analysis processing by the analysis unit 13 that is different from that in FIG. 6 will be described using FIG. 10 . In FIG. 10, it is assumed that when the product label 32 is displayed in the gaze area 35, the information processing device 20 displays a promotional video regarding the product as related information. That is, the information processing device 20 reproduces the promotion video substantially at the same time as the product label 32 is displayed in the gaze area 35, or after a predetermined period of time has elapsed since the product label 32 was displayed in the gaze area 35. do.

FIG. 10 shows the timing at which product A is displayed in the gaze area 35, that is, the timing at which the user operating the information processing device 20 pays attention to the product label of product A, and the time at which the promotional video for product A is played. showing. Furthermore, FIG. 10 shows the timing at which the user operating the information processing device 20 paid attention to the product label of product B and the time at which the promotional video for product B was played.

FIG. 10 shows that product A was displayed in the gaze area 35 at time t2, and a promotional video for product A was played between time t2 and time t3. Further, at time t4, product B is displayed in the gaze area 35, and the promotion video for product B is played between time t4 and time t5. Furthermore, it shows that the product A is displayed in the gaze area 35 at the time t6, and the promotional video regarding the product A is reproduced between the time t6 and the time t7. Further, at time t8, product B is displayed in the gaze area 35, and the promotion video for product B is played between time t8 and time t9.

In this case, the analysis unit 13 may analyze the degree of user's interest in the product according to the playback time of the promotional video. For example, the analysis unit 13 compares the period during which the promotional video for product A was played and the period during which the promotional video for product B was played. The period during which the promotional video for product A was played is the sum of the period from time t2 to time t3 and the period from time t6 to time t7. Also, the period during which the promotional video for product B was played is the sum of the period from time t4 to time t5 and the period from time t8 to time t9.

In FIG. 10, since the period during which the promotional video for product A was played is longer than the period during which the promotional video for product B was played, the analysis unit 13 may analyze that the user is interested in product A. . Alternatively, the analysis unit 13 may associate the degree of interest of the user according to the period during which the promotional video was played. The user's degree of interest may be classified into multiple levels of interest, such as, for example, strong interest, normal interest, and low interest.

Next, an example of analysis processing by the analysis unit 13, which is different from that shown in FIGS. 6 and 10, will be described with reference to FIG. FIG. 11 shows in which language the user displays the ingredient label of product A and how long the user pays attention to product A. FIG. It is also assumed that component labels in different languages are displayed as related information. Specifically, the user can change the language and visually recognize the ingredient label indicated in the related information. For example, the user can switch the language of the ingredient labels shown in the related information by selecting the English, Chinese, and Korean icons displayed on the display.

The detection period for product A shown in FIG. 11 indicates the period during which product A is displayed in display area 30 . Displaying the product A in the display area 30 may mean that part of the product A is displayed in the display area 30 . When the product A is no longer displayed in the display area 30 , the related information of the product A is also no longer displayed in the display area 30 . In other words, the related information of the product A is also displayed in the display area 30 while the product A is displayed in the display area 30 . Therefore, the analysis unit 13 may estimate that the user is viewing the component label of the product A displayed as related information while the product A is displayed in the display area 30 .

FIG. 11 shows that the user switched the language of the component labels displayed in the related information to English at time t2. FIG. 11 also shows that the user switched the language of the component label displayed in the related information to Korean at time t3. FIG. 11 also shows that the product A is displayed in the display area 30 from time t2 to time t4. In this case, the analysis unit 13 assumes that the user visually recognizes the component label written in English from time t2 to time t3, and the user visually identifies the component label written in Korean from time t3 to time t4. may be analyzed.

FIG. 11 also shows that product A is no longer displayed in display area 30 from time t4 to time t5. FIG. 11 also shows that the user switches the language of the ingredient label displayed in the related information to Chinese notation at time t5, and product A is displayed in the display area 30 from time t5 to time t6. there is FIG. 11 also shows that the display A is displayed in the display area 30 from time t7 to time t8 after the product A is not displayed in the display area 30 from time t6 to time t7. Also, FIG. 11 shows that the user switched the component label displayed in the related information to English at time t7.

In this case, the analysis unit 13 may analyze the language in which the user is interested according to how long the product A is displayed in the display area 30 after the language of the related information is switched. In FIG. 11, the periods during which the user is estimated to be visually recognizing the ingredient label of product A displayed as related information in English are the period from time t2 to time t3 and the period from time t7 to time t8. It is the total period of time up to and including FIG. 11 shows that the period during which the user is estimated to view the ingredient label of product A displayed as related information in English is the period in which the user views the ingredient label of product A in Korean. And it shows that it is longer than the period visually recognized in Chinese. From this, the analysis unit 13 may analyze that the user is interested in component labels written in English.

Next, configuration examples of the information processing device 10 and the information processing device 20 (hereinafter referred to as the information processing device 10 and the like) described in the above embodiments will be described.

FIG. 12 is a block diagram showing a configuration example of the information processing apparatus 10 and the like. Referring to FIG. 12, the information processing apparatus 10 and the like include a network interface 1201, a processor 1202, and a memory 1203. FIG. The network interface 1201 is used to communicate with other network node devices that make up the communication system. Network interface 1201 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series. Alternatively, network interface 1201 may be used to conduct wireless communications. For example, the network interface 1201 may be used for wireless LAN communication or mobile communication defined in 3GPP (3rd Generation Partnership Project).

The processor 1202 reads out software (computer program) from the memory 1203 and executes it to perform the processing of the information processing apparatus 10 and the like described using the sequence diagrams and flowcharts in the above-described embodiments. The processor 1202 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). Processor 1202 may include multiple processors.

The memory 1203 is composed of a combination of volatile memory and non-volatile memory. Memory 1203 may include storage remotely located from processor 1202 . In this case, processor 1202 may access memory 1203 via an I/O interface, not shown.

In the example of FIG. 12, memory 1203 is used to store software modules. The processor 1202 reads and executes these software modules from the memory 1203, thereby performing the processing of the information processing apparatus 10 and the like described in the above embodiments.

As described with reference to FIG. 12, each of the processors included in the information processing apparatus 10 and the like executes one or more programs including instruction groups for causing the computer to execute the algorithm described with reference to the drawings.

In the above examples, the programs can be stored and delivered to computers using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media (eg, magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories. The magnetic recording medium may be, for example, a floppy disk, magnetic tape, hard disk drive. The semiconductor memory may be mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, or RAM (Random Access Memory), for example. The program may also be delivered to the computer on various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired channels, such as wires and optical fibers, or wireless channels.

It should be noted that the present disclosure is not limited to the above embodiments, and can be modified as appropriate without departing from the scope of the present disclosure.

Some or all of the above-described embodiments can also be described in the following supplementary remarks, but are not limited to the following.

(Appendix 1)
an image acquisition unit that acquires a captured image in which an object is captured;
a display control unit that superimposes information related to the object on the captured image and displays it on a display unit;
obtaining positions of the object and the related information within the display unit, and displaying the object via the display unit using a predetermined position within the display unit and the positions of the object and the related information; and an analysis unit that analyzes the interest of a user who visually recognizes an object.
(Appendix 2)
The analysis unit
The information processing apparatus according to supplementary note 1, wherein the interest of the user is analyzed using a distance between the predetermined position and the positions of the object and the related information.
(Appendix 3)
The analysis unit
analyzing the interest of the user by further using the period during which the object is displayed in the area containing the predetermined position and the period during which the relevant information is displayed in the area containing the predetermined position; The information processing apparatus according to claim 1 or 2.
(Appendix 4)
the object includes a first region and a second region;
The analysis unit
Any of appendices 1 to 3, further using the period during which the first area is displayed and the period during which the second area is displayed in the area including the predetermined position to analyze the user's interest 1. The information processing apparatus according to 1.
(Appendix 5)
The image acquisition unit is
Acquiring a captured image in which a plurality of the objects are captured,
The display control unit
superimposing information related to each object on the captured image and displaying it on the display unit;
The analysis unit
Further using the period during which the respective objects are displayed in the areas containing the predetermined positions and the period during which the relevant information of the respective objects is displayed in the areas containing the predetermined positions, 5. The information processing device according to any one of appendices 1 to 4, which analyzes a user's interest.
(Appendix 6)
The image acquisition unit is
Acquiring a first captured image in which a first object is captured and a second captured image in which a second object is captured at a timing different from the timing when the first object is captured,
The analysis unit
A period during which the first object is displayed in an area including the predetermined position, a period during which the second object is displayed in an area including the predetermined position, and the predetermined position and a period during which the second relevant information of the second object is displayed in an area containing the predetermined position. 5. The information processing apparatus according to any one of appendices 1 to 4, wherein the interest of the user is analyzed using .
(Appendix 7)
The analysis unit
7. The user's interest is further analyzed by using the number of times at least one of the object and the related information is displayed in the area including the predetermined position. Information processing equipment.
(Appendix 8)
The analysis unit
Supplementary notes 1 to 7, wherein attributes of the user are estimated using at least one of the object displayed on the display unit, the related information displayed on the display unit, and input information input by the user. The information processing device according to any one of .
(Appendix 9)
The analysis unit
Obtaining information indicating whether or not the object included in the captured image was purchased by the user, and analyzing the reason why the user purchased the object as the user's interest. 9. The information processing apparatus according to any one of 8.
(Appendix 10)
The analysis unit
The information according to appendix 9, wherein information prompting purchase of another object related to the object purchased by the user is displayed on the display unit based on the reason why the user purchased the object. processing equipment.
(Appendix 11)
an imaging unit that images the object and generates the captured image;
11. The information processing apparatus according to any one of additional notes 1 to 10, further comprising a display unit that displays a display image in which information related to the object is superimposed on the captured image.
(Appendix 12)
Acquiring a captured image in which an object is captured,
superimposing information related to the object on the captured image and displaying it on a display unit;
Acquiring the position of the object and the related information in the display unit;
information processing for analyzing interest of a user viewing the object through the display unit using a predetermined position within the display unit and positions of the object and the related information within the display unit; Analysis method using equipment.
(Appendix 13)
Supplementary note 12, further estimating the attribute of the user using at least one of the object displayed on the display unit, the related information displayed on the display unit, and input information input by the user The analytical method described.
(Appendix 14)
obtaining information indicating whether or not the object included in the captured image has been purchased by the user;
Analyze the reason why the user purchased the object as the user's interest,
14. The method according to appendix 12 or 13, wherein, based on the reason why the user purchased the target object, information prompting purchase of another target object related to the target object purchased by the user is displayed on the display unit. analysis method.
(Appendix 15)
Acquiring a captured image in which an object is captured,
superimposing information related to the object on the captured image and displaying it on a display unit;
Acquiring the position of the object and the related information in the display unit;
a computer analyzing interest of a user viewing the object through the display unit using a predetermined position within the display unit and positions of the object and the related information within the display unit; program to run.

REFERENCE SIGNS LIST 10 information processing device 11 image acquisition unit 12 display control unit 13 analysis unit 20 information processing device 21 imaging unit 22 display unit 23 related information generation unit 30 display area 31 gaze point 32 product label 33 ingredient label 34 related information 35 gaze area

Claims (9)

an image acquisition unit that acquires a captured image in which an object is captured;
a display control unit that superimposes information related to the object on the captured image and displays it on a display unit;
The positions of the object and the related information within the display unit are acquired, and the closer the distance from the predetermined position within the display unit to the positions of the object and the related information, the closer the distance is to the position of the object and the related information, the and an analysis unit that analyzes that a user visually recognizing the object has a high interest in the object or the related information . an image acquisition unit that acquires a captured image in which an object is captured;
a display control unit that superimposes information related to the object on the captured image and displays it on a display unit;
obtaining the positions of the object and the related information in the display unit, and displaying the object in an area including a predetermined position in the display unit during a period in which the area including the predetermined position is displayed; Analyzing that, if the relevant information is displayed for a period longer than the period, the user viewing the object through the display unit is more interested in the information displayed on the object than the relevant information; When the period during which the related information is displayed in the area containing the predetermined position is longer than the period during which the object is displayed in the area containing the predetermined position, the user may display the object and an analysis unit that analyzes that there is more interest in the related information than in the information shown above . the object includes a first region and a second region;
The analysis unit
When the period during which the first area is displayed in the area containing the predetermined position is longer than the period during which the second area is displayed, the user is displayed in the second area . analyzing that the information displayed in the first area is more interesting than the information, and the period during which the second area is displayed in the area including the predetermined position is the period in which the first area is displayed 3. The information processing according to claim 2 , analyzing that the user is more interested in the information shown in the second area than the information shown in the first area, if the period is longer than the period of time indicated in the first area. Device. The image acquisition unit is
Acquiring a captured image in which a plurality of the objects are captured,
The display control unit
superimposing information related to each object on the captured image and displaying it on the display unit;
The analysis unit
Further using the period during which the respective objects are displayed in the areas containing the predetermined positions and the period during which the relevant information of the respective objects is displayed in the areas containing the predetermined positions, 4. The information processing apparatus according to claim 2 , which analyzes user interest. The image acquisition unit is
Acquiring a first captured image in which a first object is captured and a second captured image in which a second object is captured at a timing different from the timing when the first object is captured,
The analysis unit
A period during which the first object is displayed in an area including the predetermined position, a period during which the second object is displayed in an area including the predetermined position, and the predetermined position and a period during which the second relevant information of the second object is displayed in an area containing the predetermined position. 4. The information processing apparatus according to claim 2 , wherein the interest of the user is analyzed using . The analysis unit
6. The method according to any one of claims 1 to 5 , further using the number of times at least one of said object and said related information has been displayed in an area including said predetermined position to analyze said user's interest. information processing equipment. The analysis unit
3. Estimate attributes of said user using at least one of said object displayed on said display unit, said related information displayed on said display unit, and input information input by said user. 7. The information processing device according to any one of 6 . An information processing device acquires a captured image in which an object is captured,
superimposing information related to the object on the captured image and displaying it on a display unit;
Acquiring the position of the object and the related information in the display unit;
The closer the distance from the predetermined position in the display unit to the object and the related information in the display unit, the more the object or the related information of the user who visually recognizes the object through the display unit. An analysis method using an information processing device that analyzes that there is a high interest in Acquiring a captured image in which an object is captured,
superimposing information related to the object on the captured image and displaying it on a display unit;
Acquiring the position of the object and the related information in the display unit;
The closer the distance from the predetermined position in the display unit to the object and the related information in the display unit, the more the object or the related information of the user who visually recognizes the object through the display unit. A program that causes a computer to perform an analysis that is of high interest to .

Images