JP2024016454A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable presenting information according to emotions of users by taking an account of privacy protection.

SOLUTION: An information processing device comprises a control unit for performing the processing of: based on gaze information acquired by a user terminal, which is information regarding a user's line of sight toward an information provision area, and an information content provided in the information provision area, estimating a user's emotion toward the information; acquiring presentation information corresponding to the estimated emotion; and transmitting the presentation information to the user terminal.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an information processing device, an information processing method, and a program.

Conventionally, there are systems that estimate a user's emotions and present information suitable for the user based on the emotions. For example, Patent Document 1 listed below discloses a technique for presenting information based on a user's emotion and an object to which the user feels the emotion. Specifically, in Patent Document 1 listed below, when the estimated emotion is a negative emotion, information for removing the target is presented, and when the estimated emotion is a positive emotion, information is presented to maintain the emotion. is presenting.

Japanese Patent Application Publication No. 2020-109709

For example, when providing information to an unspecified number of users, such as on display devices such as electronic guide boards and electronic billboards installed in public places, the emotions of each user are estimated and displayed. Even if information is optimized, it is not desirable from the viewpoint of privacy protection if the optimized information is presented in a visible state to other users. Patent Document 1 also does not consider the case where information is provided to an unspecified number of users.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved method that can present information in accordance with the user's emotions while taking privacy protection into consideration. An object of the present invention is to provide an information processing device, an information processing method, and a program.

In order to solve the above-mentioned problems, according to one aspect of the present invention, line-of-sight information, which is information regarding the user's line of sight with respect to an information providing area, acquired by a user terminal, and information content provided in the information providing area. a control unit that performs a process of estimating the user's emotion regarding the information based on the above, a process of acquiring presentation information corresponding to the estimated emotion, and a process of transmitting the presentation information to the user terminal, An information processing device is provided.

The information providing area may be a display screen of a display device that is an object that provides information to an unspecified number of users in a public place.

The control unit may determine the emotion classification based on the user's line of sight information, and further estimate the user's emotion regarding the information based on the content of the display screen.

The control unit may calculate a gaze point of the user on the display screen based on the line-of-sight information and a captured image of the display screen acquired by the user terminal.

The control unit may estimate the user's emotion regarding the information based on the content displayed at the user's gaze point.

The presentation information corresponding to the emotion may be information that supplements the content displayed on the display screen to support the user.

The control unit may acquire information regarding content displayed on the display screen and the presentation information from a database associated with identification information of the display device.

The control unit may perform a process of setting a plurality of users into a group through authentication using a line-of-sight movement pattern on the display screen.

The control unit may determine that the authentication is successful when a line-of-sight movement pattern recognized from line-of-sight information received from the user terminal matches a line-of-sight movement pattern generated in advance.

The control unit may set the plurality of users to the same group by linking the accounts of the user terminals of the plurality of users who have been successfully authenticated.

The control unit may further set the plurality of user terminals in the same group when identification information of the display devices associated with the plurality of user terminals match.

The line-of-sight movement pattern may be the order in which the line-of-sight stays in each divided area where the display screen is divided into a plurality of areas.

The control unit sets one user terminal as a host terminal through authentication using the eye movement pattern, sets another user terminal as a client terminal through authentication using the eye movement pattern, and creates an account of the host terminal. You can set multiple users in a group by linking them.

The control unit integrates the emotions of a plurality of users set in the group to estimate the emotions of the entire group, obtains presentation information corresponding to the estimated emotions of the entire group, and uses the presentation information to create an account linked account. Control of transmission to multiple user terminals may also be performed.

According to another aspect of the present invention, in order to solve the above problems, a processor includes line-of-sight information, which is information regarding a user's line of sight with respect to an information providing area, acquired by a user terminal, and information provided in the information providing area. estimating the user's emotion with respect to the information based on the information content, obtaining presentation information corresponding to the estimated emotion, and transmitting the presentation information to the user terminal. , an information processing method is provided.

According to another aspect of the present invention, in order to solve the above-mentioned problems, a computer is provided with line-of-sight information, which is information about a user's line of sight with respect to an information providing area, which is acquired by a user terminal, and information provided in the information providing area. A process of estimating the user's emotion toward the information based on the information content, a process of acquiring presentation information corresponding to the estimated emotion, and a process of transmitting the presentation information to the user terminal. A program is provided that functions as a control unit.

As described above, according to the present invention, it is possible to present information in accordance with the user's emotions while taking privacy protection into consideration.

FIG. 1 is a diagram illustrating an overview of an information presentation system according to the present embodiment. 1 is a diagram showing an example of the configuration of an information presentation system according to the present embodiment. FIG. 2 is a block diagram showing an example of the configuration of a server 20 according to the present embodiment. FIG. 3 is a diagram illustrating acquisition of line-of-sight information and captured images according to the present embodiment. FIG. 3 is a diagram illustrating output of a gaze point according to the present embodiment. FIG. 1 is a diagram showing an example of the configuration of a user terminal 10 according to the present embodiment. FIG. 2 is a sequence diagram showing an example of the overall flow of the information presentation system according to the present embodiment. FIG. 2 is a diagram illustrating an overview of this application example. FIG. 2 is a block diagram showing an example of the configuration of a server 20A according to this application example. FIG. 7 is a diagram illustrating an example of a user notification screen of a gaze movement pattern according to this application example. FIG. 7 is a diagram illustrating recognition of a line-of-sight movement pattern according to this application example. FIG. 2 is a sequence diagram showing an example of the overall flow of the information presentation system according to this application example. FIG. 2 is a block diagram illustrating an example of the hardware configuration of an information processing device 900 according to the present embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted.

<1. Overview＞
This embodiment estimates the user's emotions when providing information to an unspecified number of users, such as a display device such as an electronic information board or an electronic billboard installed in a public place. The present invention relates to a mechanism for individually presenting information corresponding to the user's emotions.

FIG. 1 is a diagram illustrating an overview of an information presentation system according to this embodiment. In this embodiment, an example of an object that provides information to an unspecified number of users is a large display device such as an electronic information board or an electronic billboard installed in a public place. In FIG. 1, a digital signage 30 is illustrated as an example of a large-sized display device. The digital signage 30 may be free-standing, or may be installed on a wall or a pillar. For example, if the digital signage 30 is a guide board for an exhibition, the display section 31 of the digital signage 30 displays explanatory information or route guidance information for each exhibition booth. Since the display unit 31 of the digital signage 30 has a large screen size, it is easier for the user to understand the entire information or search for information by using the digital signage 30 than by looking up information on the small screen of his or her smartphone.

However, in a situation where an unspecified number of users use a display device, it is not preferable from the viewpoint of privacy protection to present information optimized according to the user's emotions in a state that is visible to other users. Conventionally, it has been difficult to optimally present information to an unspecified number of users in display devices such as electronic information boards or electronic billboards installed in public places.

Therefore, in this embodiment of the present invention, the emotion of a user searching for information is estimated using an object that provides information to an unspecified number of users, and the information corresponding to the estimated emotion is presented from the user terminal of the user. By doing this individually, information can be presented with privacy protection in mind. Thereby, the convenience of information presentation can be improved.

In the example shown in FIG. 1, the emotion of the user searching for information is estimated on the display unit 31 of the digital signage 30, and information corresponding to the user's emotion is presented from the user terminal 10. As a result, the user's privacy is protected by displaying information optimized for the user on the user terminal 10 while using the digital signage 30 with a large screen size to grasp or search for the entire information. Note that in the information presentation system according to the present embodiment, the user's emotion estimation can be performed based on gaze information obtained by the gaze sensor 140 provided in the user terminal 10. In the case of a fixed sensor that targets an unspecified number of users, such as a sensor installed in the digital signage 30, the measurement range of the line of sight is limited, and the unspecified number of users differ in user characteristics such as height, gender, or the presence or absence of vision correction. It is difficult to measure the user's line of sight. Therefore, in this embodiment, each user's line of sight is measured using a sensor that can be freely adjusted according to user characteristics, such as a sensor provided in the user terminal 10 owned by each user or a sensor attached to the user's head. By doing so, emotion estimation can be performed for all users.

FIG. 2 is a diagram showing an example of the configuration of the information presentation system according to this embodiment. As shown in FIG. 2, the information presentation system according to this embodiment includes a user terminal 10 used by a user and a server 20 (information processing device). The user terminal 10 and the server 20 are communicatively connected via a network 50 and can transmit and receive data.

The user terminal 10 is realized by any mobile terminal such as a smartphone, a mobile phone terminal, a tablet terminal, and a glasses-type wearable device. The user terminal 10 sends the user's line of sight information to the information providing area (for example, the display screen of the digital signage 30), the captured image of the display section 31 (display screen) of the digital signage 30, and the identification information of the digital signage 30 to the server 20. Send to. The user terminal 10 can read the identification information of the digital signage 30 from the QR code (registered trademark) 311 displayed on the display section 31 of the digital signage 30, for example, as shown in FIG. The QR code 311 may be attached to a location other than the display section 31 of the signage 30. Further, the QR code 311 is an example of encoded information including identification information of the digital signage 30, but is not limited thereto. The identification information of the digital signage 30 may be obtained from a one-dimensional code or a two-dimensional code. Further, the identification information of the digital signage 30 may be displayed, printed, or affixed on the digital signage 30, and the user terminal 10 may obtain this by character recognition using OCR (Optical Character Reader) technology, or the user may obtain the identification information using the user terminal. 10 may be entered manually.

The server 20 estimates the emotion of the user searching for information on the display screen of the digital signage 30 (an example of an information provision area) based on the line-of-sight information received from the user terminal 10 and the captured image of the display screen of the digital signage 30. do. Then, the server 20 acquires information corresponding to the estimated emotion and transmits the acquired information to the user terminal 10. The information corresponding to the estimated emotion is information for supporting the user searching for information, and specifically, is information that supplements the content displayed on the display screen of the digital signage 30. The user's gaze point on the display screen of the digital signage 30 is obtained from the user's line-of-sight direction included in the line-of-sight information received from the user terminal 10 and the captured image of the display screen of the digital signage 30 . Further, the server 20 can acquire the content and supplementary information displayed on the display screen of the digital signage 30 from a DB (database) for each signage, based on the identification information of the digital signage 30 received from the user terminal 10. .

In this way, the information presentation system according to the present embodiment estimates the emotion of a user who is searching for information on the digital signage 30 used by an unspecified number of users, and presents information according to the estimated emotion to the user. By performing this on the terminal 10, information presentation that takes privacy protection into consideration is realized.

Note that the digital signage 30 is an example of a display device, and the display device according to this embodiment is not limited to a digital signage. For example, the display device according to this embodiment may be a projector that projects an image onto a screen, wall, floor, or ceiling. In addition, objects that provide information to an unspecified number of users are not limited to display devices, but include, for example, information boards with venue guide maps or floor maps of commercial facilities printed on them, or explanation boards with explanatory text printed on them. , or large-sized printed matter.

<2. Configuration example of server 20>
FIG. 3 is a block diagram showing an example of the configuration of the server 20 according to this embodiment. As shown in FIG. 3, the server 20 includes a communication section 210, a control section 220, and a storage section 230.

(Communication Department 210)
The communication unit 210 functions as a transmitter and a receiver that transmits and receives information to and from other devices. The communication unit 210 can transmit and receive data to and from the user terminal 10 via the network 50, for example. The network 50 may be realized by the Internet, a telephone line network, a public line network such as a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), and the like. Further, each network may be realized by a wireless communication network such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

(Control unit 220)
The control unit 220 is configured mainly of a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like, and controls the operation of the server 20 in general.

The control unit 220 according to the present embodiment also functions as a gaze point calculation unit 221, an emotion estimation unit 222, a presentation information acquisition unit 223, and a presentation control unit 224.

Based on the information received from the user terminal 10, the gaze point calculation unit 221 calculates the gaze point of the user in the information providing area, which is an area where information is provided to an unspecified number of users. The information providing area is, for example, a display screen. In this embodiment, the gaze point calculation unit 221 calculates the user's gaze point on the display unit 31 (display screen) of the digital signage 30. Specifically, the gaze point calculation unit 221 calculates the gaze point based on the information on the user's gaze direction included in the gaze information received from the user terminal 10 and the captured image of the display unit 31 (display screen) of the digital signage 30. Then, the user's gaze point is calculated. In this embodiment, the inward camera of the user terminal 10 corresponds to the line of sight sensor 140, and the outward camera (camera 150) of the user terminal 10 obtains a captured image. FIG. 4 is a diagram illustrating acquisition of line-of-sight information and captured images according to this embodiment. As shown in FIG. 4, the direction of the user's line of sight is measured by the line of sight sensor 140 provided inside the user terminal 10, and the direction of the user's line of sight is measured by the camera 150 provided on the outside, which is a surface opposite to the surface on which the line of sight sensor 140 is provided. A captured image of the display section 31 of the digital signage 30 is obtained. A user searches for information by viewing the display section 31 of the digital signage 30 while pointing the user terminal 10 (specifically, the camera 150) toward the digital signage 30, as shown in FIG. 1, for example. During this time, the user's line of sight direction is measured by the line of sight sensor 140. Measurement of line-of-sight direction is performed continuously.

The user's line-of-sight direction detected by the line-of-sight sensor 140 may be a relative direction to the line-of-sight sensor 140, or may be an absolute direction of the line of sight in a three-dimensional space. Furthermore, the captured image obtained by the camera 150 is a visual field image that includes the user's visual field. The gaze point calculation unit 221 calculates the gaze point on the display unit 31 (display screen) shown in the captured image based on the line of sight direction and the captured image. In this embodiment, the algorithm of the gaze point calculation method is not particularly limited.

FIG. 5 is a diagram illustrating output of a gaze point according to this embodiment. The gaze point calculation unit 221 outputs a two-dimensional coordinate position P (x, y) fixed on the display unit 31 (on the display screen) of the digital signage 30, as shown in FIG. 5, as the user's gaze point. The gaze point calculation unit 221 calculates and updates the two-dimensional coordinate position P of the line of sight every unit time that is sufficient to process the user's line of sight information. In this embodiment, the update time unit is Δt, and the two-dimensional position acquired at each time is written as Pn (xn, yn) as the coordinate position after Δt×n hours have passed. In other words, when the line of sight is measured for Δt×t time, the gaze point calculation unit 221 calculates the gaze points of the user as P1 (x1, y1), P2 (x2, y2), . . . Pt (xt, yt), etc. Outputs an array of coordinate positions.

The emotion estimation unit 222 estimates the emotion of the user while searching for information. Specifically, the emotion estimating unit 222 estimates the user's emotion regarding the information based on the user's gaze point, pupil diameter, etc., and the display content (information content) on the digital signage 30. Details will be explained below.

First, the emotion estimation unit 222 uses a learned emotion estimation model generated in advance by machine learning using supervised learning to determine the emotion classification such as "interested" or "distressed" from the user's gaze information. do. Note that machine learning is not limited to supervised learning, and unsupervised learning, semi-supervised learning, etc. may also be used. Next, the emotion estimation unit 222 determines the user's more specific information, such as what the user is interested in or what is bothering him, based on the combination with the display content (attention target) of the digital signage 30. Estimate emotions. Information on the display content is obtained from the signage-specific DB 231 based on the identification information (also referred to as signage ID) of the digital signage 30 transmitted from the user terminal 10. Further, the signage ID of the digital signage 30 used by the user is not limited to being read by the user terminal 10, and the control unit 220 can read the signage ID of the digital signage 30 used by the user using the position information of the user terminal 10 transmitted from the user terminal 10 and the position information of the user terminal 10 placed in the venue. The signage ID may be specified by estimating which digital signage the user is using based on the arrangement information of each digital signage.

For example, if the emotion classification of "troubled" is estimated while observing the floor map displayed on the digital signage 30, the emotion estimation unit 222 can estimate the user's emotion such as "I'm lost." At this time, the emotion estimation unit 222 can also estimate where the user wants to go (destination), etc. by referring to the display content displayed at the gaze point. In addition, when the emotion classification of "troubled" is estimated while paying close attention to the explanatory text of the exhibit displayed on the digital signage 30, the emotion estimation unit 222 outputs the message "I don't understand the content of this explanatory text (it's difficult to explain)". , cannot read kanji, etc.)” can be estimated. Further, when the emotion classification of "interest" is estimated while observing the explanatory text of the exhibition displayed on the digital signage 30, the emotion estimation unit 222 may respond by saying, "I would like to know more detailed information about this exhibition." User emotions can be estimated. The specific user emotion in combination with the display content may be estimated using table data prepared in advance, or may be estimated using a learned model generated by machine learning.

The presentation information acquisition unit 223 acquires presentation information corresponding to the estimated user emotion. The presentation information acquisition unit 223 can acquire information from the signage-specific DB 231 stored in the storage unit 230 based on the signage ID. Note that the signage-specific DB 231 is not limited to the example stored in the storage unit 230 of the server 20, but may be stored in an external data server. For example, a data server may be prepared for each signage installed in a large number at a venue to store information regarding the signage. For example, when the digital signage 30 is a guide board for an exhibition, the signage-specific DB 231 stores detailed information about each exhibition booth or route guidance information from the position of the guide board to each exhibition booth.

When the emotion estimating unit 222 estimates the user's emotion such as "I don't understand the content of this explanatory text (it's difficult to explain, I can't read kanji, etc.)", the presentation information acquisition unit 223 displays a plain text prepared for children. The explanatory text is acquired from the signage-specific DB 231 as presentation information. Further, when the emotion estimating unit 222 estimates a user emotion such as "I'm lost," the presentation information acquisition unit 223 acquires route guidance information from the current location to each direction from the signage-specific DB 231 as presentation information. . Further, when the destination is also estimated, the presentation information acquisition unit 223 acquires detailed route guidance information from the current location to the destination from the signage-specific DB 231 as presentation information.

The presentation control unit 224 controls transmitting the presentation information acquired by the presentation information acquisition unit 223 to the user terminal 10. The information to be transmitted may include a control signal that instructs the user terminal 10 to display the presentation information on the display unit of the user terminal 10.

(Storage unit 230)
The storage unit 230 has a function of temporarily or permanently storing various data such as data related to processing of the control unit 220 and operation program files of the control unit 220. The storage unit 230 also stores a signage-specific DB 231. The signage-specific DB 231 is prepared for each of a large number of digital signages installed at a venue, for example. The signage-specific DB 231 stores information regarding signage IDs and display contents. Further, the signage-specific DB 231 may further store information on installation locations.

An example of the configuration of the server 20 according to this embodiment has been described above. Note that the configuration of the server 20 according to this embodiment is not limited to the example shown in FIG. 3. For example, the server 20 may be composed of a plurality of devices. Furthermore, in the present embodiment, a case has been described in which the functions of the server 20 (information processing device) are realized by a cloud server existing on a network, but the present invention is not limited to this, and the functions of the server 20 are realized by a cloud server existing on a network. It may be realized on hardware provided in each digital signage. That is, each digital signage may have the configuration shown in FIG. 3 (communication unit 210, control unit 220, and corresponding signage-specific DB 231).

<3. Configuration example of user terminal 10>
FIG. 6 is a diagram showing an example of the configuration of the user terminal 10 according to this embodiment. As shown in FIG. 6, the user terminal 10 includes a communication section 110, a control section 120, an operation input section 130, a line of sight sensor 140, a camera 150, a display section 160, and a storage section 170.

(Communication Department 110)
The communication unit 110 functions as a transmitter and a receiver that transmits and receives information to and from other devices. The communication unit 110 can transmit and receive data to and from the server 20 via the network 50, for example.

(Control unit 120)
The control unit 120 is configured mainly of, for example, a CPU (Central Processing Unit) and a RAM (Random Access Memory), and controls the operation of the user terminal 10 in general.

Further, the control unit 120 according to the present embodiment can control information presentation according to the user's emotions. Such control may be performed according to an application program installed on the user terminal 10. Specifically, the control unit 120 controls the camera 150 to acquire the signage ID of the digital signage 30 used by the user, and transmits it from the communication unit 110 to the server 20. Further, the control unit 120 transmits the user's line-of-sight information detected by the line-of-sight sensor 140 and the captured image of the display unit 31 (display screen) of the digital signage 30 captured by the camera 150 from the communication unit 110 to the server 20. control. Further, the control unit 120 controls displaying the presentation information corresponding to the user's emotion received from the server 20 on the display unit 160.

(Operation input unit 130)
The operation input unit 130 receives operation input from the user and outputs operation information to the control unit 120. The operation input unit 130 may be realized by a touch panel stacked on the display unit 160, or may be a physical switch.

(Gaze sensor 140)
The line of sight sensor 140 has a function of measuring the user's line of sight. For example, the line of sight sensor 140 is realized by an infrared stereo camera. Furthermore, the line-of-sight sensor 140 is mounted on the surface of the user terminal 10, that is, the surface on which the display unit 160 is provided, and measures the line-of-sight of the user holding the user terminal 10. Calibration for line of sight measurement may be performed in advance. The line of sight sensor 140 is not limited to an infrared camera, but may be a visible light camera. The line-of-sight sensor 140 can measure the line-of-sight direction, pupil diameter, and the like. The line of sight direction can be measured, for example, from the positional relationship between a predetermined reference point in the eye (for example, the inner corner of the eye or a corneal reflex) and a moving point (for example, the iris or pupil).

(Camera 150)
The camera 150 is a so-called outward-facing camera mounted on the back surface of the user terminal 10, that is, the surface opposite to the surface on which the display section 160 is provided. Camera 150 is realized by, for example, a visible light camera. The camera 150 captures an image of the display section 31 (display screen) of the digital signage 30, which is the target of the user's gaze. Furthermore, the camera 150 is used to obtain a signage ID. Specifically, a QR code including the signage ID is imaged, and the captured image of the QR code is output to the control unit 120. The control unit 120 performs image analysis of the QR code appearing in the captured image and extracts the signage ID.

(Display section 160)
The display unit 160 has a function of displaying various operation screens and presentation information corresponding to the user's emotions. The display unit 160 may be realized by a touch panel display on which the operation input unit 130 is stacked.

(Storage unit 170)
The storage unit 170 has a function of temporarily or permanently storing various data such as data related to processing of the control unit 120 and operation program files of the control unit 120.

An example of the configuration of the user terminal 10 according to the present embodiment has been described above. Note that the configuration of the user terminal 10 according to this embodiment is not limited to the example shown in FIG. 6. For example, the line-of-sight sensor 140 and the camera 150 may be provided in glasses worn on the user's head, and the line-of-sight information and the captured image (field-of-view image) may be transmitted to the user terminal 10.

<4. Operation processing>
Next, the operational processing of the information presentation system according to this embodiment will be specifically explained. FIG. 7 is a sequence diagram showing an example of the overall flow of the information presentation system according to this embodiment.

First, as shown in FIG. 7, the user terminal 10 acquires a signage ID from the captured image of the QR code displayed on the digital signage 30 (step S103). Note that a display prompting the user to read the QR code may be displayed on the digital signage 30 in order to receive information presentation according to the emotion estimation.

Next, the user terminal 10 transmits a connection request including the signage ID to the server 20 (step S106). Note that information on the destination of the connection request (the address of the server 20) can also be obtained from the QR code displayed on the digital signage 30.

Next, in response to the connection request from the user terminal 10, the server 20 associates the received signage ID with the user of the user terminal 10 (step S109). Specifically, the server 20 may associate the identification information of the user terminal 10 with the signage ID.

Next, in the user terminal 10, the line of sight sensor 140 detects the user's line of sight (step S112), and the camera 150 captures an image of the digital signage 30 (step S115). While holding the user terminal 10, the user directs his/her line of sight to the display section 31 of the digital signage 30 and views the display contents of the display section 31. Note that an explanation of how to operate and hold the user terminal 10 for emotion estimation may be displayed on the digital signage 30, or may be sent from the server 20 to the user terminal 10 in response to a connection request and notified to the user. may be done.

Subsequently, the user terminal 10 transmits the line-of-sight information and the signage captured image to the server 20 (step S118).

Next, the server 20 uses the gaze point calculation unit 221 to calculate a gaze point based on the line-of-sight information and the signage captured image (step S121).

Next, the server 20 uses the emotion estimation unit 222 to estimate the user's emotion regarding the information based on the line of sight information and the display content of the digital signage 30 (step S124).

Next, the server 20 uses the presentation information acquisition unit 223 to refer to the signage-specific DB 231 based on the signage ID associated with the user, and acquires presentation information corresponding to the user's emotion (step S127).

Next, the server 20 transmits the presentation information to the user terminal 10 using the presentation control unit 224 (step S130).

The user terminal 10 then displays the presentation information received from the server 20 on the display unit 160 (step S133). Thereby, information corresponding to the emotions of a user who is searching for information on the digital signage 30 used by an unspecified number of users can be individually presented on the user terminal 10 of the user.

An example of the operation processing of the information presentation system according to the present embodiment has been described above. Note that the flow of the operation process shown in FIG. 7 is an example, and the present invention is not limited thereto. For example, some of the processes shown in FIG. 7 may be performed in a different order from the order shown in FIG. 7, or may be performed simultaneously, or some of the processes may be skipped. . For example, the line of sight detection shown in step S112 and the imaging process shown in step S115 may be performed simultaneously for a certain period of time. Further, the transmission process shown in step S118 may be performed in parallel with the line of sight detection shown in step S112 and the imaging process shown in step S115. Further, the user terminal 10 acquires location information and transmits it to the server 20 instead of step S103, and the server 20 identifies the signage ID of the digital signage 30 used by the user from the location information of the user terminal 10. Good too.

<5. Application example>
In the embodiment described above, an unspecified number of users can use the digital signage 30 at the same time, and information according to the emotion estimation result is individually presented to each user from the user terminal 10. Here, there may be a group of friends, family, etc. among the unspecified number of users. In this case, it is desirable to estimate the emotions of the entire group and present information to the group according to the estimation results. Therefore, as an application example of the above-described embodiment, a case where a plurality of users are processed as a group will be described.

FIG. 8 is a diagram illustrating an overview of this application example. In this application example, as shown in FIG. 8, it is assumed that user X and user Y, who are a plurality of users who use the digital signage 30, are a group of friends, family, or the like. More specifically, the group setting of user X and user Y is realized by account linkage between user X's user terminal 10X and user Y's user terminal 10Y. In this application example, account linkage is performed through authentication based on each user's line-of-sight movement pattern on the display unit 31. The line-of-sight information of each user is detected in the user terminal 10 of each user, similarly to the embodiment described above.

(5-1. Configuration)
FIG. 9 is a block diagram showing an example of the configuration of the server 20A according to this application example. As shown in FIG. 9, the server 20A includes a communication section 210, a control section 220A, and a storage section 230A. Detailed description of the configurations having the same reference numerals as described with reference to FIG. 3 will be omitted here.

The control unit 220A functions as a gaze point calculation unit 221, an emotion estimation unit 222, a presentation information acquisition unit 223, a presentation control unit 224, a pattern management unit 225, a pattern recognition unit 226, and an authentication unit 227.

The pattern management unit 225 has a function of managing eye movement patterns used for group setting authentication. Specifically, the pattern management unit 225 generates a line-of-sight movement pattern used for group setting authentication, and stores it in the pattern DB 232. Since group setting authentication is performed for each signage, the pattern management unit 225 stores the generated line-of-sight movement pattern in the pattern DB 232 for each signage. The gaze movement pattern is, for example, the order in which the gaze point stays in a predetermined number of divided areas obtained by dividing the display area of the display unit 31 into equal sizes. First, the pattern management unit 225 divides the display area of the display unit 31 provided in the digital signage 30 into a predetermined number of equal sizes, and stores the range of the divided areas in the signage-specific DB 231. Next, the pattern management unit 225 randomly generates a line-of-sight movement pattern in which divided areas and in what order the gaze point should stay. Note that if the pattern management unit 225 has already generated a line-of-sight movement pattern for another group in the same signage, it generates a line-of-sight movement pattern that does not overlap with the line-of-sight movement pattern. Further, the number of divided areas is not particularly limited.

The eye movement pattern generated by the pattern management unit 225 may be presented to the user terminal 10. FIG. 10 is a diagram illustrating an example of a user notification screen of a gaze movement pattern according to this application example. As shown in FIG. 10, the display unit 160 of the user terminal 10 shows the movement order of the gaze points in the divided areas.

The pattern recognition unit 226 has a function of recognizing the user's gaze movement pattern from the user's gaze point calculated by the gaze point calculation unit 221. A detailed explanation will be given below with reference to FIG. 11.

FIG. 11 is a diagram illustrating recognition of a gaze movement pattern according to this application example. As shown in FIG. 11, in the divided areas E1 to E9 in which the display section 31 is divided into equal sizes, the pattern recognition unit 226 determines in which divided area the gaze point has stayed and in what order, and moves the line of sight. Recognize patterns. Here, staying refers to a state in which the gaze point is within a specific range. The specific range is the range of each divided area. For example, if the coordinate position Pn (xn, yn), which is the user's gaze point measured after Δt×n time has elapsed, is within the range of the divided area E1, the pattern recognition unit 226 determines that the gaze point is within the divided area E1. It is determined that the value remains at 1.

However, if the gaze point passes within a specific range of a certain divided area in a short period of time such as from Pn (xn, yn) to Pn+1 (xn+1, yn+1), the pattern recognition unit 226 recognizes that if the point of interest passes within a specific range of a certain divided area, It is not determined that the In the example shown in FIG. 11, when the gaze point moves from the divided area E7 to the divided area E5, the gaze point passes through the divided area E8, but because it only passes through the divided area E8 in a short time, the pattern recognition unit 226 It is determined that there is no retention in E8.

In the example shown in FIG. 11, the pattern recognition unit 226 detects that the gaze point stays in the order of divided area E1, divided area E4, divided area E7, divided area E5, divided area E3, divided area E6, and divided area E9. , and recognizes this order as a line-of-sight movement pattern.

The authentication unit 227 determines whether the user's line-of-sight movement pattern recognized by the pattern recognition unit 226 matches the line-of-sight movement pattern for authentication generated by the pattern management unit 225. If they match, the user's authentication is successful. In the case of authentication in response to a request to start account linkage, the authentication unit 227 sets the user terminal 10 of the user who has been successfully authenticated as the host terminal. On the other hand, in the case of authentication in response to a client authentication request, the authentication unit 227 sets the user terminal 10 of the user who has been successfully authenticated as a client terminal linked with the account of the user terminal 10 that has already been set as the host terminal. , to realize group settings. Account linkage may be performed by associating identification information of the client terminal with identification information of the host terminal, or may be performed by assigning the same group number to the host terminal and the client terminal. The authentication unit 227 may notify the user terminal 10 of the success or failure of authentication or that the accounts have been linked (that a group has been set). In the authentication of the client terminal, the authentication unit 227 may further set the signage ID to match that of the host terminal as a condition for successful authentication.

Through the above steps, group setting is performed. When group setting is performed, the emotion estimating unit 222 may integrate the emotions of each user in the group to estimate the emotion of the entire group. For example, the emotion estimation unit 222 may estimate the emotion that accounts for the majority of the emotions of each user in the group as the emotion of the entire group. Furthermore, the emotion estimation unit 222 may estimate the emotion of the host user (the user of the user terminal set as the host terminal) as the emotion of the entire group. Then, the presentation control unit 224 performs control to transmit presentation information corresponding to the emotions of the entire group to all users of the group, that is, to a plurality of user terminals 10 whose accounts are linked. This allows each user terminal 10 set in a group to view the same presentation information.

The storage unit 230A stores a signage-specific DB 231 and a pattern DB 232. The signage-specific DB 231 stores information indicating the range of divided areas in addition to information regarding the signage ID and display content. The pattern DB 232 stores one or more eye movement patterns generated for each signage. In addition, when the set group of users finishes using the digital signage 30, the control unit 220A discards the line-of-sight movement pattern used for authentication of the group from the pattern DB 232.

An example of the configuration of the server 20A according to this application example has been described above.

(5-2. Operation processing)
Next, the operational processing of the information presentation system according to this application example will be specifically explained. FIG. 12 is a sequence diagram showing an example of the overall flow of the information presentation system according to this application example.

First, as shown in FIG. 12, the user terminal 10X and the user terminal 10Y each obtain a signage ID from the captured image of the QR code displayed on the digital signage 30 (steps S203, S209), and connect the signage ID including the signage ID. The request is sent to the server 20 (steps S206, S212). In this application example, the same signage ID is acquired from the same digital signage 30 in the user terminal 10X and the user terminal 10Y, and is transmitted to the server 20A.

Next, the server 20A associates the received signage ID with each user in response to the connection requests from the user terminals 10X and 10Y (step S215). Here, the same signage ID is associated with the identification information of the user terminal 10X and the identification information of the user terminal 10Y.

Next, the user terminal 10X starts account linkage processing on the host side (step S218). For example, it is started when the user X selects a button such as "Start host-side account linkage processing" displayed on the menu screen of the user terminal 10X.

Next, the user terminal 10X transmits a request to start account linkage to the server 20A (step S221).

Next, the server 20A uses the pattern management unit 225 to generate a gaze movement pattern in response to the account linkage start request (step S224).

Next, the server 20A transmits the generated line-of-sight movement pattern to the user terminal 10X (step S227).

Next, the user terminal 10X displays the received gaze movement pattern on the display unit 160 (step S233). A display example is as shown in FIG. After confirming the line-of-sight movement pattern, user

Next, in the user terminal 10X, the line of sight sensor 140 detects the line of sight of the user X (step S233), and the camera 150 captures an image of the digital signage 30 (step S236).

Subsequently, the user terminal 10X transmits the line-of-sight information and the signage captured image of the user X to the server 20A (step S239).

Next, the server 20A uses the gaze point calculation unit 221 to calculate the gaze point of the user X based on the user X's line of sight information and the signage captured image (step S242).

Next, the server 20A uses the pattern recognition unit 226 to recognize the eye movement pattern of the user X based on the calculated gaze point, and determines whether the eye movement pattern matches the eye movement pattern generated for authentication using the authentication unit 227. If they match, user X is authenticated as a host (step S245). Specifically, if the eye movement patterns match, the authentication unit 227 sets the user terminal 10X as the host terminal.

Next, the server 20A notifies the user terminal 10X that the host authentication has been successful (step S248).

The user terminal 10X displays on the display unit 160 to notify the user X that the host authentication has been successful (step S251). At this time, the user terminal 10X may display information on the display unit 160 that guides the user who wants to participate in the group to share the eye movement pattern for authentication. User X shows and shares a display of a gaze movement pattern for authentication, such as the one shown in FIG. 10, with user Y who is present with him, for example.

In the user terminal 10Y, account linkage processing on the client side is started (step S254). For example, when the user Y selects a button such as "Start client-side account linkage processing" displayed on the menu screen of the user terminal 10Y, the user terminal 10Y starts the account linkage process. When user Y confirms the gaze movement pattern shared by user X (host), he performs an operation to start account linkage processing, and then starts gaze input on the display unit 31 of the digital signage 30. The user terminal 10Y performs the following processing as account linkage processing.

First, the user terminal 10Y detects the line of sight of the user Y using the line of sight sensor 140 (step S257), and further images the digital signage 30 using the camera 150 (step S260).

Next, the user terminal 10Y transmits a client authentication request including the line-of-sight information of the user Y and the signage captured image to the server 20A (step S263).

Next, the server 20A uses the gaze point calculation unit 221 to calculate the gaze point of the user Y based on the user Y's line of sight information and the signage captured image (step S266).

Next, the server 20A uses the pattern recognition unit 226 to recognize the line-of-sight movement pattern of the user Y based on the calculated gaze point, and the authentication unit 227 performs client authentication (step S269). Specifically, the authentication unit 227 determines whether the eye movement pattern of user Y matches the eye movement pattern generated for authentication. Next, if the line-of-sight movement patterns match and the signage ID associated with user Authenticate as a client (set user terminal 10Y as a client terminal). Setting a client terminal means linking an account with a host terminal, thereby realizing a group setting for multiple users. The server 20A may notify the user terminals 10X and 10Y that the client authentication has been successful (that the group has been set).

The group setting operation processing according to this application example has been specifically explained above. In this application example, by performing authentication using gaze movement patterns for group settings (account linkage), the hassle of operations such as inputting passcode strings is eliminated, and the user can move his gaze in a predetermined order. Authentication becomes possible using a simpler method. Note that the flow of the operation process shown in FIG. 12 is an example, and the present invention is not limited thereto. Further, after the group setting is performed, as described above, the server 20A integrates the emotion estimation results of each user in the group to estimate the emotion of the entire group. Then, the server 20A performs control to obtain information corresponding to the estimated emotions of the entire group and to individually present the information to each user of the group. Thereby, each group-set user can view the same presentation information on each user terminal 10.

<6. Hardware configuration example>
Next, the hardware configurations of the user terminal 10, server 20, and server 20A according to this embodiment will be described. The above operations are realized through cooperation between software and hardware described below.

FIG. 13 is a block diagram showing an example of the hardware configuration of the information processing device 900 according to this embodiment. The information processing device 900 is an example of a hardware configuration applied to the user terminal 10, server 20, and server 20A according to this embodiment.

The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface 9. 07 and , an input device 908, an output device 909, a storage device 910, a drive 911, and a communication device 913.

The CPU 901 functions as an arithmetic processing device and a control device, and controls overall operations within the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus and the like. The CPU 901, ROM 902, and RAM 903 work together to realize the control unit 120 of the user terminal 10, the control unit 220 of the server 20, or the control unit 220A of the server 20A.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the operator to input information, such as a mouse, keyboard, touch panel, button, switch, or microphone, and an input control device that generates an input signal based on the input by the operator and outputs it to the CPU 901. It consists of circuits, etc. By operating the input device 908, the operator of the information processing device 900 can input various data to the information processing device 900 and instruct processing operations. The input device 908 implements the operation input unit 130 of the user terminal 10.

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker. The output device 909 realizes the display unit 160 of the user terminal 10.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 realizes the storage unit 170 of the user terminal 10, the storage unit 230 of the server 20, or the storage unit 230A of the server 20A.

The drive 911 is a reader/writer for storage media, and is externally attached to the information processing apparatus 900. The drive 911 reads information recorded on a removable storage medium 912 such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs it to the RAM 903 . The drive 911 can also write information to a removable storage medium 912.

The communication device 913 is a communication interface configured with a communication device and the like for performing communication. The communication device 913 realizes the communication unit 110 of the user terminal 10 or the communication unit 210 of the server 20 or server 20A.

Note that the hardware configuration of the information processing device 900 is not limited to the configuration shown in FIG. 13. For example, the information processing apparatus 900 does not need to include the communication device 913 when communicating via a connected external communication device. Further, the information processing apparatus 900 does not need to include the input device 908 or the output device 909, for example. Further, for example, part or all of the configuration shown in FIG. 13 may be realized by one or more ICs (Integrated Circuits).

<7. Supplement>
Although preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea stated in the claims. It is understood that these also naturally fall within the technical scope of the present invention.

For example, the calculation of the gaze point performed by the gaze point calculation unit 221 of the server 20 may be performed by the control unit 120 of the user terminal 10. The user terminal 10 calculates the gaze point on the display screen of the digital signage 30 based on the user's line of sight information and the captured image of the digital signage 30, and sends the calculation result (information on the coordinate position corresponding to the gaze point) to the server. It may be sent to 20.

Further, the account linkage according to the application example is not limited to the presentation of information corresponding to user emotions, but can also be utilized when it is desired to link multiple user accounts. For example, the account linkage according to the application example can also be applied to communication between users of an interpersonal game.

Also, one or more computer programs for causing hardware such as a CPU, ROM, and RAM built in the user terminal 10, server 20, or server 20A to perform the functions of the user terminal 10, server 20, or server 20A. can also be created. Also provided is a computer readable storage medium having the one or more computer programs stored thereon.

10 user terminal 110 communication unit 120 control unit 130 operation input unit 140 line of sight sensor 150 camera 160 display unit 170 storage unit 20, 20A server 210 communication unit 220, 220A control unit 221 gaze point calculation unit 222 emotion estimation unit 223 presentation information acquisition unit 224 Presentation control unit 225 Pattern management unit 226 Pattern recognition unit 227 Authentication unit 230, 230A Storage unit 231 DB by signage
232 Pattern DB
30 Digital signage 31 Display section 311 QR code

Claims (16)

A process of estimating the user's emotion toward the information based on line-of-sight information, which is information regarding the user's line of sight toward the information-providing area, acquired by a user terminal, and information content provided in the information-providing area;
a process of acquiring presentation information corresponding to the estimated emotion;
a process of transmitting the presentation information to the user terminal;
An information processing device comprising a control unit that performs. The information processing device according to claim 1, wherein the information providing area is a display screen of a display device that is an object that provides information to an unspecified number of users in a public place. The information processing device according to claim 2, wherein the control unit determines the emotion classification based on the user's line of sight information, and further estimates the user's emotion regarding the information based on the content of the display screen. The information according to claim 2 or 3, wherein the control unit calculates the user's gaze point on the display screen based on the line-of-sight information and a captured image of the display screen acquired by the user terminal. Processing equipment. The information processing device according to claim 2 or 3, wherein the control unit estimates the user's emotion regarding the information based on the content displayed at the user's gaze point. 4. The information processing apparatus according to claim 2, wherein the presentation information corresponding to the emotion is information that supplements content displayed on the display screen to support the user. The information processing device according to claim 2 or 3, wherein the control unit acquires information regarding the content displayed on the display screen and the presentation information from a database associated with identification information of the display device. The information processing apparatus according to claim 2 or 3, wherein the control unit performs a process of setting a plurality of users into a group through authentication using a line-of-sight movement pattern on the display screen. The information processing device according to claim 8, wherein the control unit determines that the authentication is successful when a gaze movement pattern recognized from gaze information received from a user terminal matches a gaze movement pattern generated in advance. The information processing apparatus according to claim 9, wherein the control unit sets the plurality of users to the same group by linking the accounts of the user terminals of the plurality of users who have been successfully authenticated. The information processing apparatus according to claim 10, wherein the control unit further sets the plurality of user terminals in the same group when identification information of the display devices associated with the plurality of user terminals match. The information processing device according to claim 8, wherein the line-of-sight movement pattern is an order of line-of-sight retention in each divided area where the display screen is divided into a plurality of areas. The control unit includes:
Setting one user terminal as a host terminal by authentication using the eye movement pattern,
Setting a plurality of users into a group by setting another user terminal as a client terminal through authentication using the eye movement pattern and linking it to the account of the host terminal;
The information processing device according to claim 8. The control unit includes:
Integrate the emotions of multiple users set in a group to estimate the emotions of the entire group,
Obtain presentation information that corresponds to the estimated emotions of the entire group,
Controls sending of the presented information to multiple user terminals linked to accounts,
The information processing device according to claim 8. The processor
Estimating the user's emotion regarding the information based on gaze information, which is information regarding the user's line of sight toward the information providing area, acquired by a user terminal, and information content provided in the information providing area;
obtaining presentation information corresponding to the estimated emotion;
transmitting the presentation information to the user terminal;
information processing methods, including computer,
A process of estimating the user's emotion toward the information based on line-of-sight information, which is information regarding the user's line of sight toward the information-providing area, acquired by a user terminal, and information content provided in the information-providing area;
a process of acquiring presentation information corresponding to the estimated emotion;
a process of transmitting the presentation information to the user terminal;
A program that functions as a control unit that performs.

Images