WO2020032400A1

WO2020032400A1 - Method for determining emotion of user and device therefor

Info

Publication number: WO2020032400A1
Application number: PCT/KR2019/008400
Authority: WO
Inventors: 남재현; 조안나; 최태준
Original assignee: 주식회사 룩시드랩스
Priority date: 2018-08-09
Filing date: 2019-07-09
Publication date: 2020-02-13
Also published as: KR102186580B1; KR20200017797A

Abstract

The present invention, according to one embodiment of the present invention, provides a method for determining the emotion of a user, and a device therefor. The method comprises the steps of: providing, via a head-mounted display (HMD) device configured to acquire first gaze data and first brain wave data of a user, a first user interface including a first visual object; acquiring, via the HMD device, the first brain wave data, of the user, corresponding to the first visual object associated with the first gaze data; determining the acquired first brain wave data to be reference biometric information for determining the emotion of the user; providing, via the HMD device, a second user interface including a second visual object; acquiring, via the HMD device, second gaze data, of the user, for the second visual object, and acquiring, via the HMD device, second brain wave data, of the user, corresponding to the second visual object associated with the acquired second gaze data; and generating emotion information associated with the second visual object on the basis of the reference biometric information and the second brain wave data.

Description

Method for determining emotion of user and apparatus for same

The present invention relates to a method for determining an emotion of a user and an apparatus therefor.

In general, a head-mounted display (HMD) device (hereinafter, referred to as an 'HMD device') is formed in a structure that can be worn on a user's head so that the user can experience a spatial and temporal experience similar to the reality. A display device that provides an image about virtual reality (VR), augmented reality (AR), or mixed reality (MR).

The HMD device is composed of a body formed in the form of goggles (goggle) to be worn on the user's eyes, and wearing portion formed in the form of a band to be connected to the body to fix the body to the user's head. In this case, as a means for outputting an image related to virtual reality, augmented reality and / or mixed reality, a display device is provided, a portable terminal device such as a smartphone is installed, or a display device connected to a PC such as a monitor is mounted. do.

Recently, in order to analyze the effect of viewing of an image related to a user's virtual reality, augmented reality, or mixed reality on the user, the HMD device includes a biometric sensor for acquiring the biometric information of the user. While watching the video, signals such as EEG, gaze, electrocardiogram, safety, and EMG of the user may be acquired. The electronic device connected to the HMD device may analyze the signal acquired from the biometric sensor to determine how the user feels while watching the image.

However, when acquiring the brain waves of the user while watching the image related to the virtual reality, augmented reality, or mixed reality as described above, the brain waves of the user are acquired for each specific scene of the image watched by the user, and the acquired brain waves are analyzed. Since the emotion of the user is determined, it is difficult to predict in advance what emotion the user has with respect to the specific visual object gazed by the user.

The problem to be solved by the present invention is to provide a method and apparatus for determining the emotion of the user to solve the above problems.

Specifically, the present invention is to provide a method and apparatus for determining the emotion of a user with respect to a particular visual object by using the user's gaze data and brain wave data.

The objects of the present invention are not limited to the objects mentioned herein, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, the present invention provides a method and apparatus for determining the emotion of the user. The method includes providing a first user interface comprising a first visual object via a Head-Mounted Display (HMD) device configured to obtain a user's first gaze data and first brain wave data; Acquiring first brain wave data of the user corresponding to the first visual object associated with the first gaze data through the HMD device; Determining the obtained first brain wave data as reference biometric information for determining the user emotion; Providing a second user interface comprising a second visual object via the HMD device; Acquiring second gaze data of the user with respect to the second visual object through the HMD device, and acquiring second brain wave data of the user corresponding to a second visual object associated with the acquired second gaze data; ; And generating emotion information about the second visual object based on the reference biometric information and the second brain wave data.

The device provides a first user interface comprising a first visual object via a Head-Mounted Display (HMD) device configured to obtain a user's first gaze data and first brain wave data, and wherein the HMD Acquire first brain wave data of the user corresponding to the first visual object associated with the first gaze data through the device, and determine the obtained first brain wave data as reference biometric information for determining the user emotion; And providing a second user interface including a second visual object through the HMD device, and obtaining the second gaze data of the user with respect to the second visual object through the HMD device. Obtain second brainwave data of the user corresponding to a second visual object associated with the Generating emotion information about the second visual object based on the reference biometric information and the second brain wave data.

Details of other embodiments are included in the detailed description and drawings.

The present invention generates emotion information about a specific visual object on which a user gazes by using reference biometric information determined based on the gaze data and brain wave data of the user, thereby making it possible for the user to gaze at various visual objects. The emotion of the user can be predicted.

Effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a schematic diagram illustrating an apparatus for determining a user emotion according to an embodiment of the present invention.

2 is a schematic diagram illustrating an HMD device in which an image output device is mounted according to an embodiment of the present invention.

3 is a schematic diagram illustrating an electronic device connected to an HMD device and an image output device according to an embodiment of the present invention.

4 is a schematic diagram illustrating an HMD device having a display and an electronic device connected to the HMD device according to an embodiment of the present invention.

FIG. 5 is a schematic flowchart illustrating a method for generating emotion information of a user with respect to a specific visual object in a user emotion determination apparatus according to an embodiment of the present invention.

6A, 6B, 6C, and 6D are exemplary diagrams for describing a method for determining reference biometric information in a user emotion determining apparatus according to an embodiment of the present invention.

7A and 7B are exemplary diagrams for describing a method for determining reference biometric information in a user emotion determination apparatus according to an embodiment of the present invention.

8A and 8B are exemplary diagrams for describing a method for determining reference biometric information in a user emotion determining apparatus according to an embodiment of the present invention.

9 is an exemplary diagram for describing a method of generating emotion information indicating a user's emotion with respect to a specific visual object in a user emotion determination apparatus according to an embodiment of the present invention.

10A and 10B are exemplary diagrams for describing a method for correcting reference biometric information in a user emotion determining apparatus according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only the present embodiments to make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

Although the first, second, etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Like reference numerals refer to like elements throughout.

Each of the features of the various embodiments of the present invention may be combined or combined with each other, partly or wholly, and various technically interlocking and driving are possible as one skilled in the art can fully understand, and each of the embodiments may be independently implemented with respect to each other. It may be possible to carry out together in an association.

In the present specification, the apparatus for determining a user emotion according to an embodiment of the present invention is not limited, and may include any device configured to acquire a gaze of a user and obtain biometric information such as brain waves of the user. For example, the user emotion determination device may include a sensor for contacting / wearing a part of a user's body such as a head mounted display (HMD) device, a headset, a smart ring, a smart watch, an earset, an earphone, and obtaining a user's biometric information. The image output device may output an image related to a device, a virtual reality, an augmented reality, or a mixed reality, and an electronic device managing the same. For example, when the HMD device has a display, the user emotion determining device may include the HMD device and the electronic device. Here, the biometric information may represent various signals generated from the user's body according to the user's conscious and / or unconscious (eg, breathing, heart rate, metabolism, etc.) behavior such as pulse, blood pressure, brain wave, and the like.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram illustrating an apparatus for determining a user emotion according to an embodiment of the present invention. Referring to FIG. 1, the user emotion determining apparatus may include an HMD device 100 and may include at least one of an image output device 200 and an electronic device 300. For example, when the HMD device 100 includes a display, the image output device 200 may not be included.

The HMD device 100 is mounted on the user's head to provide the user with an image of virtual reality, augmented reality, or mixed reality so that the user can have a spatial and temporal experience that is similar to reality, and at the same time, the user's eyes and brain waves, etc. It may be a complex virtual experience device that can acquire physical information and sense physical, cognitive, and emotional changes of a user who is undergoing a virtual experience.

In detail, the HMD device 100 is formed in a structure that is wearable on a user's head, and the HMD device 100 is implemented to perform image processing for virtual reality, augmented reality, or mixed reality inside the HMD device 100 or the HMD device 100. The image output device 200 providing an image related to the virtual reality, the augmented reality, or the mixed reality is mounted on a part of the), and the image output device 200 performs image processing for the virtual reality in the mounted image output device 200. Can be.

When the HMD device 100 has a display, one side of the display may be disposed to face the user's face so that when the user wears the HMD device 100, the user may check an image about virtual reality, augmented reality, or mixed reality. Can be.

In various embodiments, an accommodating space 101 may be formed in a portion of the HMD device 100 to accommodate the image output device 200. When the image output apparatus 200 is accommodated in the accommodation space 101, one surface of the image output apparatus 200 (for example, one surface on which the display of the image output apparatus 200 is located) may be disposed to face the user's face. For example, the image output device 200 may include a portable terminal device such as a smartphone, a tablet PC, or the like, or a portable monitor capable of outputting an image provided from the PC by being connected to the PC.

One side of the HMD device 100 may be formed with at least one sensor (not shown) for acquiring gaze data and brain wave data of the user. The at least one sensor may include a gaze tracking sensor for tracking the eyes of the user and an EEG sensor for measuring the brain waves of the user. In various embodiments, the at least one sensor is formed at a position where the user's eyes or face can be photographed or a position where the user's skin can be contacted, so that the user's eye or face is photographed when the user wears the HMD device 100. The photographed image may be analyzed to acquire eye gaze data of the user or may be in contact with the skin of the user to measure electroencephalography (EEG) of the user to acquire brain wave data.

The HMD device 100 obtains the gaze data of the user and the brain wave data of the user according to a request of the image output device 200 or the electronic device 300, and obtains the obtained gaze data and the brain wave data from the image output device 100 or the like. It may transfer to the electronic device 300.

In detail, when the display includes the HMD device 100, the first user interface including the first visual object to obtain the first gaze data and the first brain wave data of the user through the display of the HMD device 100. I can display it. The first visual object or the first user interface may be provided from the electronic device 300. The first user interface is a user interface for rewarded content that provides a reward service for a specific behavior of a user in response to a first visual object, and the first visual object is used for rewarded content constituting the first user interface. It may include at least one or a combination of graphic objects, such as related text, images, icons, and the like. The rewarded content may include advertisements, surveys, events, problem solving or games. For example, the first user interface may be a user interface for playing an advertisement, a user interface for a survey, a user interface for event participation, a user interface for problem solving, or a game play. The first visual object may include an image representing an advertisement target included in a user interface for playing an advertisement, text representing a survey content included in a user interface for a survey, and event contents included in a user interface for event participation. It may be an image representing a text, an image, a game related character, an item, a background, etc. included in a user interface for game play. The specific behavior of the user includes any one of various behaviors that satisfy the conditions for acquiring the reward data. The specific behavior of the user may be gazing at an image corresponding to an advertisement target included in the user interface for a specific time, or participating in a survey or event. And obtaining certain items or scores in the game.

The HMD device 100 may obtain first brain wave data of the user for the first visual object associated with the first gaze data, and transmit the acquired first brain wave data to the electronic device 300. According to various embodiments of the present disclosure, the HMD device 100 collects and stores the first EEG data of the user for the first visual object associated with the first gaze data independently, such as without being connected to the electronic device 300. When connected to the 300, the stored first EEG data may be transmitted to the electronic device 300.

The HMD device 100 may display a second user interface including a second visual object to obtain second gaze data and second brain wave data of the user for generating emotion information of the user with respect to a specific visual object. I can display it. The second visual object or the second user interface may be provided from the electronic device 300. The HMD device 100 may obtain second gaze data of the user for the second visual object, and acquire second brain wave data of the user corresponding to the second visual object associated with the obtained second gaze data. The HMD device 100 may transmit the acquired second brain wave data to the electronic device 300. According to various embodiments of the present disclosure, the HMD device 100 collects and stores second brain wave data of a user for a second visual object associated with second eye data independently, such as a state in which the electronic device 300 is not connected to the electronic device 300. When connected to the 300, the stored second EEG data may be transmitted to the electronic device 300.

In various embodiments, the HMD device 100 may display a third user interface including a plurality of visual objects on the display to improve the accuracy of the reference biometric information. The plurality of visual objects or the third user interface may be provided from the electronic device 300. The HMD device 100 may obtain third gaze data of the user, and obtain third brain wave data of the user corresponding to the third visual object associated with the third gaze data among the plurality of visual objects. The HMD device 100 may transmit the acquired third brain wave data to the electronic device 300. According to various embodiments of the present disclosure, the HMD device 100 collects and stores the third brainwave data of the user for the third visual object associated with the third gaze data independently, such as when not connected to the electronic device 300. When connected to the 300, the stored third EEG data may be transmitted to the electronic device 300.

According to various embodiments, when the image output apparatus 200 is mounted on the HMD device 100, the image output apparatus 200 may display a first user interface including a first visual object through a display of the image output apparatus 200. I can display it. The image output apparatus 200 obtains the first gaze data of the user for the first visual object through the HMD device 100, and the first brain wave of the user for the first visual object associated with the obtained first gaze data. Data may be obtained through the HMD device 100. The image output device 200 may transmit the acquired first brain wave data to the electronic device 300.

The image output device 200 may display a second user interface including a second visual object on a display. The image output device 200 obtains the user's second gaze data for the second visual object through the HMD device 100, and the user's second brain wave for the second visual object associated with the acquired second gaze data. Data may be obtained through the HMD device 100. The image output device 200 may transmit the acquired second brain wave data to the electronic device 300.

In various embodiments, the image output apparatus 200 may display a third user interface including a plurality of visual objects on a display in order to improve the accuracy of the reference biometric information. The image output apparatus 200 obtains the third gaze data of the user through the HMD device 100, and the third brain wave data of the user corresponding to the third visual object associated with the third gaze data among the plurality of visual objects. Can be obtained. The image output device 200 may transmit the acquired third brain wave data to the electronic device 300.

The electronic device 300 may be electrically connected to the HMD device 100 or / and the image output device 200, and provide a user interface to the HMD device 100 or the image output device 200. The electronic device 300 may receive an EEG data from the HMD device 100 or receive an emotion information of a user from the image output device 200 and display an interface screen indicating the same through a display. The electronic device 300 may communicate with the HMD device 100 or the image output device 200 through wired communication.

When the HMD device 100 has a display, the electronic device 300 provides the HMD device 100 with a first user interface including the first visual object, and corresponds to the first visual object from the HMD device 100. The first EEG data may be received. The electronic device 300 may determine the received first brain wave data as reference biometric information for determining user emotion.

In detail, the electronic device 300 may request the first gaze data of the user from the HMD device 100 and receive the first gaze data from the HMD device 100. The electronic device 300 may determine whether the received first gaze data corresponds to the first visual object. The first gaze data may include first location coordinates according to the gaze of the user. The electronic device 300 may determine whether the mapped coordinates correspond to the position coordinates of the first visual object by mapping the first position coordinates with the coordinates of the specific position on the display of the HMD device 100. For example, when the mapped coordinates correspond to any one of the position coordinates of the pixels constituting the first visual object, the electronic device 300 may determine that the mapped coordinates correspond to the position coordinates of the first visual object. . When the mapped coordinates correspond to the position coordinates of the first visual object, the electronic device 300 may determine that the first gaze data corresponds to the first visual object. When the first gaze data corresponds to the first visual object, the electronic device 300 may determine whether a specific action of the user with respect to the first visual object is detected. When the specific action is detected, the electronic device 300 may determine whether compensation data is provided according to the result of the specific action, and request the first EEG data of the user from the HMD device 100 when the compensation data is provided. When the first brain wave data is received from the HMD device 100, the electronic device 300 may determine the received first brain wave data as reference biometric information for determining user emotion. For example, when reward data corresponding to a positive reward (eg, item or score acquisition) is provided, the electronic device 300 may determine first emotion based on the first EEG data received from the HMD device 100. It may be determined as the first reference biometric information. If reward data corresponding to a negative reward (eg, deduction of score, etc.) is provided or no reward data is provided (eg, no item or score is obtained), the electronic device 300 receives the first EEG received from the HMD device 100. The data may be determined as second reference biometric information for determining negative emotions. For example, positive emotions may include joy, happiness, joy, and the like, and negative emotions may include sadness, anger, suffering, and the like. In the above, the case where the reward data is provided or not provided is described, but in various embodiments, when the reward data includes a positive reward (eg, earning a score) or a negative reward (eg, subtracting a score, etc.), a positive reward is provided. When the compensation data including the compensation data is provided, the electronic device 300 determines the first emotion based on the first EEG data acquired through the HMD device 100 when the compensation data including the positive compensation is provided. The information may be determined, and when the reward data including the negative reward is provided, the first brain wave data acquired through the HMD device 100 may be determined as the second reference biometric information for determining the negative emotion. In various embodiments, a plurality of reference biometric information may be determined to determine various emotions of the user. For example, user's joy, happiness, interest, enjoyment, excitement, comfort, indifference, surprise, sadness, fear, anger, worried, nervous, confusion, anxiety, hatred, hate, annoyed, tired, cool, chilly, depressed, etc. Reference biometric information for determining an emotion of the patient may be determined.

The electronic device 300 may provide a second user interface including a second visual object to the HMD device 100 and receive second brain wave data corresponding to the second visual object from the HMD device 100. . The electronic device 300 may generate emotion information about the second visual object based on the received second brain wave data and reference biometric information. In detail, the electronic device 300 requests the second gaze data of the user from the HMD device 100, and when the second gaze data is received from the HMD device 100, the received second gaze data is the second. It may be determined whether it corresponds to a visual object. For example, the second gaze data may include second location coordinates with respect to the user's gaze, and the second visual object may include text representing various contents such as an advertisement, a game, etc., to determine what emotion the user has; It may include at least one or a combination of graphic objects, such as an image, an icon. The electronic device 300 may determine whether the mapped coordinates correspond to the position coordinates of the second visual object by mapping the second position coordinates with the coordinates of the specific position on the display of the HMD device 100. When the second location coordinates correspond to the location coordinates of the second visual object, the electronic device 300 may determine that the second gaze data corresponds to the second visual object. If it is determined that the second gaze data corresponds to the second visual object, the electronic device 300 requests the second EEG data of the user from the HMD device 100, and receives the second EEG data from the HMD device 100. Can be. The electronic device 300 checks reference biometric information that matches the received second brain wave data, and transmits emotion information (eg, positive emotion or negative emotion, etc.) related to the confirmed reference biometric information for the second visual object. It may be generated by the user's emotion information. For example, when the reference biometric information that matches the second brain wave data is the first reference biometric information corresponding to the positive emotion, the electronic device 300 may generate the positive emotion as emotion information on the second visual object. have. According to various embodiments of the present disclosure, when the reference biometric information that matches the second EEG data is the second reference biometric information corresponding to the negative emotion, the electronic device 300 may generate the negative emotion as emotion information on the second visual object.

The electronic device 300 may store the generated emotion information or display an interface screen indicating the emotion information on a display.

According to various embodiments of the present disclosure, the second visual object may include at least one graphic object corresponding to a category having a feature or a subject unrelated to the first visual object or not related to each other. For example, when the first visual object relates to a game, the second visual object may be a graphic object related to an advertisement. In addition, when the first visual object relates to the adventure game, the second visual object may be a graphic object related to the role playing game. In addition, when the first visual object relates to a car advertisement, the second visual object may be a graphic object for a cosmetic advertisement.

As described above, the present invention can predict the emotion of the user with respect to the second visual object even if the first visual object corresponds to a category having a feature or theme unrelated to the second visual object or not related to the second visual object.

The electronic device 300 provides the HMD device 100 with a third user interface including a plurality of visual objects to improve the accuracy of the reference biometric information, and the user of the plurality of visual objects from the HMD device 100. And third brain wave data corresponding to a third visual object associated with the third gaze data of the. The electronic device 300 may correct the reference biometric information by using the received third brain wave data. In detail, the electronic device 300 may obtain third gaze data of the user through the HMD device 100 and identify a third visual object related to the third gaze data among the plurality of visual objects. For example, the electronic device 300 obtains third gaze data representing third location coordinates of the user's gaze through the HMD device 100, and displays the acquired third location coordinates and the HMD device 100. The third visual object corresponding to the mapped coordinate may be confirmed by mapping specific coordinates of the image. Through this, it is possible to confirm which visual object of the plurality of visual objects the user's gaze looks at. If the gaze of the user is determined to gaze at the third visual object, the electronic device 300 requests the third EEG data of the user from the HMD device 100 and receives the third EEG data from the HMD device 100. Can be. The electronic device 300 may correct the reference biometric information by using the received third brain wave data. According to various embodiments of the present disclosure, when the user's selection input for the third visual object is received as well as the user's eyes, the electronic device 300 acquires the third EEG data of the user through the HMD device 100 when the selection input is received. The reference biometric information may be corrected based on the obtained third brain wave data.

According to various embodiments, when the image output device 200 is mounted on the HMD device 100, the electronic device 300 may include a first visual object including a first visual object to obtain first gaze data and first brain wave data of a user. The user interface may be provided to the image output apparatus 200. The electronic device 300 obtains the first gaze data of the user for the first visual object through the HMD device 100, and the first brain wave data of the user for the first visual object associated with the obtained first gaze data. Can be obtained through the HMD device 100. The electronic device 300 may determine the obtained first brain wave data as reference biometric information for determining an emotion of a user. In detail, the electronic device 300 provides the first user interface including the first visual object to the image output device 200, and when the first user interface is displayed through the image output device 200, the HMD device 100. ) May request the first gaze data of the user. For example, the electronic device 300 may transmit an image including the first user interface to the image output device 200. The image may be a virtual reality image, an augmented reality image, or a mixed reality image. When the first gaze data is received from the HMD device 100, the electronic device 300 may determine whether the received first gaze data corresponds to the first visual object. When the first gaze data corresponds to the first visual object, the electronic device 300 checks whether a specific action of the user with respect to the first visual object is detected, and if a specific action is detected, compensation according to the result of the detected specific action You can check if the data is provided. For example, the electronic device 300 may analyze the first user interface displayed on the image output device 200 to determine whether a specific action is detected and whether compensation data is provided according to a result of the specific action. The electronic device 300 requests the first EEG data of the user from the HMD device 100 when the compensation data is provided, and the user receives the received first EEG data when the first EEG data is received from the HMD device 100. The reference biometric information for determining an emotion may be determined.

The electronic device 300 outputs a second user interface including a second visual object to obtain second gaze data and second brain wave data of the user for generating emotion information of the user with respect to a specific visual object. The user's second gaze data with respect to the second visual object may be acquired through the HMD device 100. The electronic device 300 obtains, through the HMD device 100, second brain wave data of the user corresponding to the second visual object associated with the second gaze data, based on the reference biometric information and the acquired second brain wave data. Emotion information representing the emotion of the user with respect to the second visual object may be generated. In detail, the electronic device 300 provides a second user interface including the second visual object to the image output device 200, and when the second user interface is displayed through the image output device 200, the HMD device 100. ) May request the second gaze data of the user. For example, the electronic device 300 may transmit an image including the second user interface to the image output device 200. The image may be a virtual reality image, an augmented reality image, or a mixed reality image. When the second gaze data is received from the HMD device 100, the electronic device 300 may determine whether the received second gaze data corresponds to the second visual object. When the second gaze data corresponds to the second visual object, the electronic device 300 requests the second EEG data of the user from the HMD device 100, and receives the second EEG data when the second EEG data is received from the HMD device 100. 2 Reference biometric information matching the EEG data can be confirmed. The electronic device 300 may generate emotion information related to the identified reference biometric information as emotion information of the user with respect to the second visual object. For example, when the reference biometric information that matches the second brain wave data is the first reference biometric information corresponding to the positive emotion, the electronic device 300 may generate the positive emotion as emotion information on the second visual object. have. According to various embodiments of the present disclosure, when the reference biometric information that matches the second EEG data is the second reference biometric information corresponding to the negative emotion, the electronic device 300 may generate the negative emotion as emotion information on the second visual object.

In various embodiments, the electronic device 300 may provide the image output device 200 with a third user interface including a plurality of visual objects to improve the accuracy of the reference biometric information. The electronic device 300 obtains third brain wave data of a user corresponding to a third visual object among a plurality of visual objects through the HMD device 100, and obtains reference biometric information using the acquired third brain wave data. You can correct it. Specifically, the electronic device 300 provides a third user interface including a plurality of visual objects to the image output device 200, and when the third user interface is displayed through the image output device 200, the HMD device 100. ) May request the third eye data of the user. For example, the electronic device 300 may transmit an image including the third user interface to the HMD device 100. The image may be a virtual reality image, an augmented reality image, or a mixed reality image. When the third gaze data is received from the HMD device 100, the electronic device 300 may determine which of the plurality of visual objects the received third gaze data corresponds to. Through this, it is possible to confirm which visual object of the plurality of objects the user's gaze looks at. For example, when the received third gaze data corresponds to the third visual object, the electronic device 300 requests the third EEG data of the user from the HMD device 100, and the third EEG from the HMD device 100. When the data is received, the reference biometric information may be corrected using the received third brain wave data. According to various embodiments of the present disclosure, when the user's selection input for the third visual object is received, the electronic device 300 acquires the third brain wave data of the user through the HMD device 100 when the selection input is received, and the acquired third The reference biometric information may be corrected based on the EEG data.

Hereinafter, an embodiment in which the image output apparatus 200 generates the emotion information of the user to determine the emotion of the user will be described.

In various embodiments, the image output apparatus 200 may determine the first EEG data obtained from the HMD device 100 as reference biometric information for determining the emotion of the user. In detail, the image output apparatus 200 may request the first gaze data of the user from the HMD device 100 and receive the first gaze data from the HMD device 100. The image output apparatus 200 may determine whether the received first gaze data corresponds to the first visual object. When the first gaze data corresponds to the first visual object, the image output apparatus 200 may determine whether a specific action of the user with respect to the first visual object is detected. When a specific action is detected, the image output device 200 may determine whether compensation data is provided according to the result of the specific action, and request the first EEG data of the user from the HMD device 100 when the compensation data is provided. . When the first EEG data is received from the HMD device 100, the image output device 200 may determine the received first EEG data as reference biometric information for determining user emotion.

The image output apparatus 200 may display, on a display, a second user interface including a second visual object to obtain gaze data and brain wave data of the user for generating emotion information of the user for a specific visual object. have. The image output apparatus 200 obtains the second gaze data of the user for the second visual object through the HMD device 100, and the second gaze of the user corresponding to the second visual object associated with the obtained second gaze data. EEG data can be obtained. The image output apparatus 200 may generate emotion information indicating a user's emotion with respect to the second visual object based on the reference biometric information and the acquired second brain wave data. In detail, the image output apparatus 200 requests the second gaze data of the user from the HMD device 100, and when the second gaze data is received from the HMD device 100, the received second gaze data is a second visual object. It can be determined whether or not to correspond to. If it is determined that the second gaze data corresponds to the second visual object, the image output apparatus 200 requests the second EEG data of the user from the HMD device 100 and receives the second EEG data from the HMD device 100. can do. The image output apparatus 200 confirms reference biometric information that matches the received second brain wave data, and transmits emotion information (eg, positive emotion or negative emotion) related to the confirmed reference biometric information to the second visual object. The emotion information of the user may be generated. The image output device 200 may transmit the generated emotion information to the electronic device 300.

In various embodiments, the image output apparatus 200 may display a third user interface including a plurality of visual objects on a display in order to improve the accuracy of the reference biometric information. When the third user interface is provided, the image output apparatus 200 may obtain third eye gaze data of the user through the HMD device 100. The image output apparatus 200 obtains, through the HMD device 100, third brain wave data of a user corresponding to a third visual object associated with third eye data among a plurality of visual objects, and uses the third brain wave data. The reference biometric information can be corrected. According to various embodiments of the present disclosure, when the user's selection input for the third visual object is received as well as the user's eyes, the image output apparatus 200 acquires the third brain wave data of the user through the HMD device 100 when the selection input is received. The reference biometric information may be corrected based on the third EEG data obtained.

In the above description, the HMD device 100 provides the EEG data corresponding to the visual object related to the gaze data of the user to the image output device 200 or the electronic device 300. However, in various embodiments, the HMD device 100 May generate reference biometric information, generate emotion information of the user using the reference biometric information, correct the reference biometric information, and the like. These operations may be performed in a state where the electronic device 300 is not connected. The HMD device 100 stores the generated emotion information and then stores the emotion information stored when the electronic device 300 is connected to the electronic device 300. You can also pass

2 is a schematic diagram illustrating an HMD device in which an image output device is mounted according to an embodiment of the present invention. Hereinafter, an embodiment in which the image output apparatus 200 generates emotion information of a user will be described with reference to FIG. 2.

1 and 2, the HMD device 100 includes a communication interface 110, a memory 130, a sensor 140, and a processor 150, and the image output device 200 includes a communication interface 210. ), A display 220, a memory 230, and a processor 240.

First, referring to the HMD device 100, the communication interface 110 of the HMD device 100 connects the HMD device 100 to communicate with an external device. For example, the communication interface 110 may connect with the image output device 200 and the electronic device 300 using wired / wireless communication. The wireless communication may include wireless fidelity (WiFi), Bluetooth, low power Bluetooth (BLE), Zigbee, near field communication (NFC), magnetic secure transmission, radio frequency (RF), or body area network. (BAN) and the like, and wired communication includes a universal serial bus (USB), a high definition multimedia interface (HDMI), a standard standard232 (RS-232), a power line communication, or a plain old telephone service (POTS). It may include at least one. The communication interface 110 receives a request for the gaze data and the brain wave data of the user from the image output device 200 or the electronic device 300, and receives the gaze data and the brain wave data of the user obtained through the sensor 140. The image output device 200 may be transferred to the electronic device 300.

The memory 130 may store sensing information (eg, gaze data and brain wave data) obtained through the sensor 140.

The sensor 140 may include a gaze tracking sensor 142 for tracking the eyes of the user and an EEG sensor 144 for measuring the brain waves of the user. For example, the gaze tracking sensor 142 may include a camera sensor for capturing a user's eyes or face, and may analyze gaze captured by the camera sensor to output gaze data indicating a position coordinate of the gaze of the user. have. In various embodiments, a variety of methods may be used to track the user's eyes, and in addition to image analysis, a contact lens method (reflected light of a mirror-embedded contact lens or a gaze tracking method using a magnetic field of a coil-embedded contact lens) or a sensor may be attached. The eye of the user may be tracked using a method (a line of sight tracking method using an electric field according to eye movement by attaching a sensor around the eye). In addition, the EEG sensor 144 may include at least one electroencephalogram (EEG) sensor and / or magnetoencephalography (MEG) and a near-infrared spectrometer (NIRS). The EEG sensor may measure the electrical / optical frequency that varies according to the EEG of various frequencies generated from the body part of the user or the activation state of the brain and output the EEG data.

The processor 150 is operatively connected with the communication interface 110, the memory 130, and the sensor 140, and is acquired through the sensor 140 at the request of the image output device 200 or the electronic device 300. The visual gaze data and the brain wave data may be transmitted to the image output device 200 or the electronic device 300. In detail, the processor 150 receives the gaze data acquired through the gaze tracking sensor 142 when the request for the gaze data of the user is received from the image output device 200 or the electronic device 300. Alternatively, the information may be transferred to the electronic device 300. When the processor 150 receives a request for the brain wave data of the user from the image output apparatus 200 or the electronic device 300, the processor 150 outputs the brain wave data acquired through the brain wave sensor 144 to the image output apparatus 200 or the electronic apparatus. Can be passed to 300.

Referring to the image output apparatus 200, the communication interface 210 of the image output apparatus 200 connects the image output apparatus 200 to communicate with an external device. For example, the communication interface 210 connects to the HMD device 100 using wired / wireless communication, and receives a signal from the HMD device 100 indicating that the image output device 200 is mounted on the HMD device 100. Can be received. The communication interface 210 may transmit a request for gaze data and brain wave data of the user to the HMD device 100, and receive gaze data and brain wave data of the user from the HMD device 100. According to various embodiments of the present disclosure, the communication interface 210 may transmit emotion information indicating the emotion of the user with respect to a specific visual object to the electronic device 300.

The display 220 may display various contents (eg, text, images, videos, icons, banners, or symbols) to the user. In detail, the display 220 displays a virtual reality image, an augmented reality image, or a mixed reality image. The display 220 includes a first user interface including a first visual object, a second user interface including a second visual object, or a plurality of visuals. The virtual reality image representing the third user interface including the objects may be displayed.

The memory 230 may store various data used to display an image or determine an emotion of a user. In detail, the memory 230 may store a virtual reality image and an application, a program, or a widget used to display the virtual reality image. In addition, the memory 230 may store various data used to display a virtual reality image representing a user interface including a specific visual object. The memory 230 may store gaze data and brain wave data of the user received from the HMD device 100.

The processor 240 is operatively connected to the communication interface 210, the display 220, and the memory 130, and may perform various commands for determining the emotion of the user with respect to a specific visual object.

In detail, the processor 240 displays a first user interface including the first visual object to acquire first gaze data and first brain wave data of the user for determining the emotion of the user with respect to a specific visual object. ) Can be displayed. In various embodiments, the first visual object or the first user interface including the first visual object may be provided from the electronic device 300.

The processor 240 acquires first gaze data of the user through the HMD device 100, and transmits the first EEG data of the user corresponding to the first visual object associated with the acquired first gaze data. Can be obtained through The processor 240 may determine the obtained first brain wave data as reference biometric information for determining an emotion of a user.

In detail, when the first user interface is displayed through the display 220, the processor 240 requests the first gaze data including the first position coordinates of the user's gaze from the HMD device 100, and the HMD device ( The first gaze data may be received from 100. The processor 240 may map the first position coordinates of the user's gaze with the specific coordinates on the display 220 and determine whether the mapped coordinates correspond to the position coordinates of the first visual object. For example, if the mapped coordinates correspond to any one of the position coordinates of the pixels constituting the first visual object, the processor 240 may determine that the mapped coordinates correspond to the position coordinates of the first visual object. When the mapped coordinates correspond to the position coordinates of the first visual object, the processor 240 may determine that the first gaze data corresponds to the first visual object. When the first gaze data corresponds to the first visual object, the processor 240 may determine whether a specific action of the user with respect to the first visual object is detected. When the specific behavior is detected, the processor 240 may determine whether compensation data is provided according to the result of the detected specific behavior, and when the compensation data is provided, the processor 240 may request the first EEG data of the user from the HMD device 100. For example, the processor 240 may analyze the virtual reality image displayed on the display 220 to determine whether a specific behavior is detected or compensation data based on a result of the specific behavior is provided. When the first brain wave data of the user is received from the HMD device 100, the processor 240 may determine the received first brain wave data as reference biometric information for determining the emotion of the user. For example, when reward data corresponding to a positive reward (eg, item or score acquisition) is provided, the processor 240 may determine the positive emotion of the user based on the first EEG data received from the HMD device 100. It can be determined as the first reference biometric information for. If compensation data corresponding to the negative compensation is provided or no compensation data is provided, the processor 240 may determine the first EEG data received from the HMD device 100 as second reference biometric information for determining a negative emotion of the user. have.

The processor 240 displays a second user interface including a second visual object to obtain second gaze data and second brain wave data of the user for generating emotion information of the user for a specific visual object. Can be displayed through For example, the second visual object may be content such as an advertisement or a game for identifying what emotion the user has. In various embodiments, the second visual object or the second user interface including the second visual object may be provided from the electronic device 300.

When the second user interface is displayed through the display 220, the processor 240 acquires second eye data of the user for the second visual object through the HMD device 100 and is associated with the obtained second eye data. The second brain wave data of the user corresponding to the second visual object may be acquired. The processor 240 may generate emotion information indicating a user's emotion with respect to the second visual object based on the reference biometric information and the acquired second brain wave data. Specifically, when the second user interface is displayed through the display 220, the processor 240 requests the second gaze data including the second position coordinates of the user's gaze from the HMD device 100, and the HMD device ( When the second gaze data is received from 100, it may be determined whether the received second gaze data corresponds to the second visual object. For example, the processor 240 may map the second position coordinates and specific coordinates on the display 220, and determine whether the mapped coordinates correspond to the position coordinates of the second visual object. When the mapped coordinates correspond to the position coordinates of the second visual object, the processor 240 requests the second EEG data of the user from the HMD device 100, and receives the second EEG data of the user from the HMD device 100. Can be. The processor 240 may identify reference biometric information that matches the received second brain wave data. For example, the processor 240 compares the second EEG data with the first EEG data corresponding to the reference biometric information, and when the frequency difference value between the two EEG data is less than or equal to a preset threshold, the reference living body coincides with the second EEG data. The information can be confirmed with the first EEG data. The processor 240 may generate emotion information related to the identified reference biometric information as emotion information on the second visual object. For example, when the emotion information related to the identified reference biometric information is a positive emotion, the processor 240 may generate the positive emotion as emotion information of the user with respect to the second visual object. If the emotion information related to the identified reference biometric information is a negative emotion, the processor 240 may generate the negative emotion as emotion information of the user with respect to the second visual object.

In various embodiments, the processor 240 may store the generated emotion information in the memory 230 or transfer the generated emotion information to the electronic device 300.

In various embodiments, the processor 240 may display, via the display 220, a third user interface including a plurality of visual objects to improve the accuracy of the reference biometric information. The plurality of visual objects may include images related to various emotional state information. For example, the plurality of visual objects may include images related to positive emotional state information, or may include images related to positive emotional state or negative emotional state information. According to various embodiments, the third user interface including the plurality of visual objects or the plurality of visual objects may be provided from the electronic device 300.

When the second user interface is displayed through the display 220, the processor 240 acquires third eye data of the user for the third visual object from among the plurality of visual objects through the HMD device 100, and obtains Third brain wave data of the user corresponding to the third visual object associated with the third gaze data may be acquired. The processor 240 may correct the reference biometric information by using the acquired third EEG data. In detail, when the third user interface is displayed through the display 220, the processor 240 requests the third gaze data including the third position coordinates of the user's gaze from the HMD device 100, and the HMD device ( When the third gaze data is received from 100, the received third gaze data may identify which visual object among the plurality of visual objects. For example, the processor 240 may map the third position coordinates and specific coordinates on the display 220, and determine whether the mapped coordinates correspond to which of the plurality of visual objects. When the mapped coordinates correspond to the position coordinates of the third visual object, the processor 240 requests the third EEG data of the user from the HMD device 100, and receives the third EEG data of the user from the HMD device 100. Can be. The processor 240 may correct the reference biometric information by using the received third brain wave data.

According to various embodiments of the present disclosure, the processor 240 sets various emotional state information corresponding to each of the plurality of visual objects, and the emotional state information corresponding to the third visual object associated with the third gaze data of the user among the plurality of visual objects. You can check. For example, the plurality of objects may include a visual object corresponding to a positive emotion or a visual object corresponding to a negative emotion. If the identified emotional state information is a positive emotion, the processor 240 may correct the first reference biometric information determined to determine the positive emotion using the third brain wave data. When the identified emotional state information is a negative emotion, the processor 240 may correct the second reference biometric information determined to determine the negative emotion using the third brain wave data.

3 is a schematic diagram illustrating an electronic device connected to an HMD device and an image output device according to an embodiment of the present invention. Hereinafter, an embodiment in which the electronic device 300 generates user emotion information will be described with reference to FIG. 3.

1, 2, and 3, the electronic device 300 may include a communication interface 310, a display 320, a memory 330, and a processor 340.

The communication interface 310 connects the electronic device 300 to communicate with an external device. For example, the communication interface 310 may be connected to the HMD device 100 in which the image output device 200 is mounted and the image output device 200 using wired communication. In detail, the communication interface 310 may transmit a request for gaze data and brain wave data of the user to the HMD device 100, or receive gaze data and brain wave data of the user from the HMD device 100. The communication interface 310 may transmit a request to the image output apparatus 200 to display a user interface including a corresponding visual object together with a specific visual object. In various embodiments, the communication interface 310 may transmit the virtual reality image including the corresponding user interface to the image output apparatus 200 with the request.

The display 320 may display various contents to the user. In detail, the display 320 may display information indicating gaze data and brain wave data of the user received from the HMD device 100 through the communication interface 310. The display 320 may display emotion information generated by the processor 340. In various embodiments, the display 320 may simultaneously display a user interface displayed through the image output apparatus 200 with the image output apparatus 200.

The memory 330 may store various data used to determine the emotion of the user. In detail, the memory 330 may store gaze data and brain wave data of the user received from the HMD device 100. The memory 330 may store a virtual reality image representing a user interface including a specific visual object or store emotion information generated by the processor 340.

The processor 340 is operatively connected to the communication interface 310, the display 320, and the memory 330, and may perform various commands for generating emotion information of the user with respect to a specific visual object.

In detail, the processor 340 provides the first user interface including the first visual object to the image output apparatus 200, and when the first user interface is displayed through the image output apparatus 200, the HMD apparatus 100. The first gaze data of the user may be requested. When the first gaze data including the first location coordinates of the user's gaze is received from the HMD device 100, the processor 340 may determine whether the received first gaze data corresponds to the first visual object. When the first gaze data corresponds to the first visual object, the processor 340 may request the user's first brain wave data from the HMD device 100. When the first brain wave data is received from the HMD device 100, the processor 340 may determine the received first brain wave data as reference biometric information for determining the emotion of the user. In detail, the processor 340 maps the first positional coordinates of the user's gaze and the specific positional coordinates on the display 220 of the image output apparatus 200 to determine whether the mapped coordinates correspond to the positional coordinates of the first visual object. You can judge. When the mapped coordinates correspond to the position coordinates of the first visual object, the processor 340 may determine whether a specific behavior of the user with respect to the first visual object is detected. For example, the processor 340 may analyze a virtual reality image being displayed on the display 220 of the image output apparatus 200 in real time to determine whether a specific action is detected. When the virtual reality image is game content, the first user interface including the first visual object may be a user interface for game play including a game item. The processor 340 may analyze the user interface being displayed through the image output apparatus 200 to determine whether an action of acquiring a game item is detected. When a specific action is detected, the processor 340 may determine whether compensation data is provided according to the result of the specific action, and request the first EEG data of the user from the HMD device 100 when the compensation data is provided. For example, when an action of acquiring a game item is detected, the processor 340 checks whether reward data, such as assigning the acquired game item to a user's game account, is provided, and when the reward data is provided, the HMD device ( 100) the first brain wave data of the user may be requested. When the first brain wave data is received from the HMD device 100, the processor 340 may determine the received first brain wave data as reference biometric information corresponding to a user's positive or negative emotion. For example, when reward data corresponding to a positive reward is provided, the first EEG data obtained from the HMD device 100 may be determined as first reference biometric information for determining a positive emotion. When reward data corresponding to a negative reward is provided, the first EEG data obtained from the HMD device 100 may be determined as second reference biometric information for determining a negative emotion.

The processor 340 provides a second user interface including a second visual object to the image output apparatus 200, and when the second user interface is displayed through the image output apparatus 200, the user to the HMD apparatus 100. May request a second line of sight data. When the second gaze data including the second position coordinates of the user's gaze is received from the HMD device 100, the processor 340 may determine whether the received second gaze data corresponds to the second visual object. When the second gaze data corresponds to the second visual object, the processor 340 may request the user's second brain wave data from the HMD device 100. When the second brain wave data is received from the HMD device 100, the processor 340 may generate emotion information indicating the emotion of the user with respect to the second visual object based on the reference biometric information and the second brain wave data. Specifically, the processor 340 maps the second position coordinates of the user's gaze with the specific position coordinates on the display 220 of the image output apparatus 200 to determine whether the mapped coordinates correspond to the position coordinates of the second visual object. You can judge. When the mapped coordinates correspond to the position coordinates of the second visual object, the processor 340 requests the second EEG data of the user from the HMD device 100, and receives the second EEG data from the HMD device 100. Reference biometric information that matches the second EEG data may be confirmed. The processor 340 may generate emotion information (eg, positive emotion or negative emotion) related to the identified reference biometric information as emotion information of the user with respect to the second visual object.

In various embodiments, the processor 340 may provide the image output apparatus 200 with a third user interface including a plurality of visual objects to improve the accuracy of the reference biometric information. When the third user interface is displayed through the image output device 200, the processor 340 requests the third gaze data of the user from the HMD device 100, and the third position of the gaze of the user from the HMD device 100. Third line of sight data including coordinates may be received. The processor 340 requests the HMD device 100 for third brain wave data of the user corresponding to the third visual object associated with the third gaze data from among the plurality of visual objects, and the third brain wave from the HMD device 100. Data can be received. The processor 340 may correct the reference biometric information by using the received third brain wave data.

In detail, the processor 340 maps the third position coordinates of the user's gaze and the specific position coordinates on the display 220 of the image output apparatus 200 so that the mapped coordinates of any one of the plurality of visual objects. It may be determined whether it corresponds to the position coordinate. When the mapped coordinates correspond to the position coordinates of the third visual object, the processor 340 may determine that the gaze of the user gazes at the third visual object. For example, if it is determined that the gaze of the user gazes at the third visual object of the plurality of visual objects, the processor 340 may determine that the user has a positive emotion with respect to the third visual object. The processor 340 requests the third EEG data of the user from the HMD device 100, and when the third EEG data is received from the HMD device 100, the processor 340 corresponds to a positive emotion using the received third EEG data. The first reference biometric information may be corrected. For example, the processor 340 may correct the frequency value of the EEG data corresponding to the first reference biometric information to match the frequency value of the third EEG data.

4 is a schematic diagram illustrating an HMD device having a display and an electronic device connected to the HMD device according to an embodiment of the present invention. Hereinafter, an embodiment in which the electronic device 300 generates emotion information of a user will be described with reference to FIG. 4.

1 to 4, the HMD device 100 includes a communication interface 110, a memory 130, a sensor 140, and a processor 150, as described above with reference to FIG. 2, and includes a specific visual object. It may further include a display 120 for displaying a user interface that includes. As described above with reference to FIG. 3, the electronic device 300 may include a communication interface 310, a display 320, a memory 330, and a processor 340.

First, referring to the HMD device 100, the communication interface 110 of the HMD device 100 is connected to the electronic device 300 using wired / wireless communication to include a specific visual object from the electronic device 300. The user may receive a request for displaying a user interface or receive a request for gaze data and brain wave data of the user, and may transmit the gaze data and brain wave data of the user acquired through the sensor 140 to the electronic device 300. According to various embodiments, the communication interface 110 may receive a virtual reality image representing a user interface including a visual object from the electronic device 300.

The display 120 may display various contents to the user. In detail, the display 10 displays a virtual reality image, an augmented reality image, a mixed reality image, and the like, the first user interface including a first visual object, a second user interface including a second visual object, or a plurality of The virtual reality image representing the third user interface including the visual objects may be displayed.

The memory 130 may store various data for displaying an image. In detail, the memory 130 may store a virtual reality image and an application, a program, or a widget used to display the virtual reality image. In addition, the memory 130 may store various data used to display a virtual reality image representing a user interface including a specific visual object. The memory 140 may store the gaze data and the brain wave data of the user output through the sensor 140.

As described above with reference to FIG. 2, the sensor 140 may include a gaze tracking sensor 142 for tracking the eyes of the user and an EEG sensor 144 for measuring the brain waves of the user.

The processor 150 is operatively connected to the communication interface 110, the display 120, the memory 130, and the sensor 140, and at the request of the electronic device 300, a specific visual object through the display 120. It may display a user interface including or transmit the gaze data and the brain wave data acquired through the sensor 140 to the electronic device 300.

In detail, the processor 150 receives a request for displaying a user interface including a specific visual object from the electronic device 300 through the communication interface 110, and in response to the request, displays the user interface through the display 120. I can display it. For example, the user interface that includes a particular visual object may be a first user interface that includes a first visual object, a second user interface that includes a second visual object, or a third user interface that includes a plurality of visual objects. have.

The processor 150 receives a request for gaze data of the user from the electronic device 300, obtains gaze data of the user associated with a specific visual object through the gaze tracking sensor 140, and obtains the gaze Data may be transmitted to the electronic device 300. For example, the gaze data may be first gaze data, second gaze data, or third gaze data.

The processor 150 receives a request for the brain wave data of the user from the electronic device 300, obtains the brain wave data of the user corresponding to the visual object associated with the gaze data through the brain wave sensor 144 according to the request, The acquired EEG data may be transmitted to the electronic device 300. For example, the EEG data may be first EEG data, second EEG data, or third EEG data.

Referring to the electronic device 300, the communication interface 310 of the electronic device 300 may be connected to the HMD device 100 having a display through wired communication as described above with reference to FIG. 3. In detail, the communication interface 310 may transmit a request for gaze data and brain wave data of the user to the HMD device 100, or receive gaze data and brain wave data of the user from the HMD device 100. The communication interface 310 may transmit a request to the HMD device 100 to display a user interface including the visual object together with the specific visual object. In various embodiments, the communication interface 310 may transmit the virtual reality image including the corresponding user interface to the HMD device 100 with the request.

The display 320 may display various contents to the user as described above with reference to FIG. 3. In detail, the display 320 may display information indicating gaze data and brain wave data of the user received from the HMD device 100 through the communication interface 310. The display 320 may display emotion information generated by the processor 340. In various embodiments, the display 320 may simultaneously display a user interface displayed through the image output apparatus 200 with the image output apparatus 200.

The memory 330 may store various data used to determine the emotion of the user as described above with reference to FIG. 3. In detail, the memory 330 may store gaze data and brain wave data of the user received from the HMD device 100. The memory 330 may store a virtual reality image representing a user interface including a specific visual object or store emotion information generated by the processor 340.

The processor 340 is operatively connected to the communication interface 310, the display 320, and the memory 330 as described above with reference to FIG. 3, and may execute various instructions for generating user's emotion information on a specific visual object. Can be done.

In detail, the processor 340 transmits a request to the HMD device 100 to provide a first user interface including the first visual object, and through the display 120 of the HMD device 100, the first user interface. If is displayed, the first gaze data of the user may be requested to the HMD device 100. When the first gaze data of the user is received from the HMD device 100, the processor 340 determines whether the received first gaze data corresponds to the first visual object, and when the first gaze data corresponds to the first visual object, the HMD The device 100 may request first brain wave data of the user. When the first brain wave data is received from the HMD device 100, the processor 340 may determine the received first brain wave data as reference biometric information for determining the emotion of the user.

The processor 340 may provide the HMD device 100 with a second user interface including a second visual object to obtain second gaze data and second brain wave data for determining a user's emotion. When the second user interface is displayed through the display 120 of the HMD device 100, the processor 340 requests the second gaze data of the user from the HMD device 100, and the second user interface from the HMD device 100. The line of sight data can be received. The processor 340 may determine whether the received second gaze data corresponds to the second visual object, and if the second gaze data corresponds to the second visual object, request the second brain wave data of the user from the HMD device 100. have. When the second brain wave data is received from the HMD device 100, the processor 340 may generate emotion information of the user for the second visual object based on the reference biometric information and the second brain wave data.

In various embodiments, the processor 340 may provide the HMD device 100 with a third user interface including a plurality of visual objects to improve the accuracy of the reference biometric information. When the third user interface is displayed through the display 120 of the HMD device 100, the processor 340 requests the third gaze data of the user from the HMD device 100, and the third user interface is transmitted from the HMD device 100. The line of sight data can be received. The processor 340 requests the HMD device 100 for third brain wave data of the user corresponding to the third visual object associated with the third gaze data from among the plurality of visual objects, and the third brain wave from the HMD device 100. Data can be received. The processor 340 may correct the reference biometric information by using the received third brain wave data.

Referring to FIG. 5, a user interface including a first visual object may be provided through the HMD device 100 configured to acquire first gaze data and first brain wave data of a user (S500). For example, the user emotion determination device may include the HMD device 100 having the display or the HMD device 100 and the electronic device 300 having the image output device 200 mounted thereon when the HMD device 100 does not have a display. ) May be included. First EEG data of the user corresponding to the first visual object associated with the first gaze data is obtained through the HMD device 100 (S510), and the acquired first EEG data is used as reference biometric information for determining the user emotion. It may be determined (S520). A second user interface including a second visual object may be provided through the HMD device 100 (S530). The second gaze data of the user for the second visual object may be obtained through the HMD device 100, and the second brain wave data of the user corresponding to the second visual object associated with the obtained second gaze data may be obtained. (S540). Emotion information representing the emotion of the user with respect to the second visual object may be generated based on the reference biometric information and the acquired second brain wave data (S550).

6A, 6B, 6C, and 6D are exemplary diagrams for describing a method for determining reference biometric information in a user emotion determining apparatus according to an embodiment of the present invention. In the following, a user interface including a visual object may be displayed through the display 120 of the HMD device 100 or the display 220 of the image output device 200, and the operations performed below may be performed by the electronic device 300. The processor 340, the processor 240 of the image output device 200, or the processor 150 of the HMD device 100 may be used.

Referring to FIG. 6A, the apparatus for determining user emotion may display a user interface 600 for game content that matches whether the number of first visual objects 602 is odd or even. The apparatus for determining emotion of the user may display the user interface 600 for a preset first time and determine whether the first gaze data obtained by acquiring first gaze data of the user corresponds to the first visual object 602. have. In various embodiments, the user emotion determining apparatus may obtain first gaze data associated with any one of the plurality of visual objects included in the user interface 600.

Referring to FIG. 6B, the apparatus for determining a user emotion may include a user interface for selecting even or odd numbers for a preset second time after the acquired first gaze data corresponds to the first visual object 602. 610 may be displayed. For example, the user interface 610 may include a first icon 612 for selecting that the number of first visual objects 602 is even and a number for selecting an odd number of first visual objects 602. 2 icon 614 and the text 616 indicating the remaining time of the second time.

The apparatus for determining emotion of the user determines whether a specific behavior of the user with respect to the first visual object 602 is detected, and if specific behavior with respect to the first visual object 602 is detected, compensation data based on the result of the specific behavior is provided. You can check if it works. For example, when a user input for selecting the second icon 614 is received, the user emotion determining apparatus determines that a specific action is detected in the received user input, and positive reward data according to the user input as shown in FIG. 6C. The user interface 620 for providing the information may be displayed. Referring to FIG. 6C, when the number of first visual objects 602 corresponding to a user input is an odd number, the user emotion determining apparatus includes an image 622 for indicating that the 'odd number' selected by the user is the correct answer and a positive value thereof. The user interface 620 including text (eg, +1000 won) 624 for displaying reward data corresponding to the reward may be displayed.

When the compensation data is provided, the user emotion determining apparatus may obtain first brain wave data of the user and determine the obtained first brain wave data as reference biometric information for determining the emotion of the user. For example, when the user interface 620 for providing compensation data corresponding to the positive reward is displayed as shown in FIG. 6C, the user emotion determining apparatus determines that the user has positive emotion, and obtains the obtained first brain wave data. The first reference biometric information for determining positive emotion may be determined.

In various embodiments, when a user input for selecting the first icon 612 is received, the user emotion determining apparatus may display a user interface 630 for providing negative compensation data according to the user input as shown in FIG. 6D.

Referring to FIG. 6D, when the number of visual objects 602 corresponding to a user input is an even number, the user emotion determination apparatus corresponds to an image 632 and a negative compensation for indicating that the 'even' selected by the user is an incorrect answer. The user interface 630 may include a text (eg, 1,000 won) 634 to indicate reward data. When the user interface 630 is displayed, the user emotion determining apparatus may obtain first brain wave data of the user, and determine the obtained first brain wave data as reference biometric information for determining the emotion of the user. For example, the user emotion determining apparatus determines that the user has a negative emotion when the reward data corresponding to the negative reward is provided, and a second criterion for determining the negative emotion based on the first EEG data obtained when the reward data is provided. It can be determined by biometric information.

7A and 7B are exemplary diagrams for describing a method for determining reference biometric information in a user emotion determination apparatus according to an embodiment of the present invention. In the following, a user interface including a visual object may be displayed through the display 120 of the HMD device 100 or the display 220 of the image output device 200, and the operations performed below may be performed by the electronic device 300. The processor 340, the processor 240 of the image output device 200, or the processor 150 of the HMD device 100 may be used.

Referring to FIG. 7A, the apparatus for determining user emotion may display a user interface 700 for game content for obtaining a specific item. The user interface 700 may include a first visual object 702 related to game content. For example, the first visual object 702 may be a graphic object that composes a game environment to obtain a specific game item.

The apparatus for determining emotion of a user may acquire first gaze data of a user and determine whether the acquired first gaze data corresponds to the first visual object 702. According to various embodiments of the present disclosure, the apparatus for determining user emotion may acquire first gaze data associated with any one of a plurality of visual objects included in the user interface 700. If the acquired first gaze data corresponds to the first visual object 702, the user emotion determining apparatus may determine whether a specific behavior of the user with respect to the first visual object 702 is detected. For example, the specific action may be an action in which a game character performs a game quest related to the first visual object 702 to obtain a specific game item. When the specific behavior is detected, the user emotion determination apparatus may determine whether compensation data is provided according to the result of the specific behavior, and acquire first brain wave data of the user when the compensation data is provided. In detail, when the execution of the game quest is confirmed, the user emotion determination device determines whether the game quest is completed, and when the game quest is completed, as shown in FIG. 7B, the user interface 710 for providing a game item as a reward for completing the game quest. I can display it. Referring to FIG. 7B, the user interface 710 may include an image 712 for indicating a game item acquired according to completion of a game quest, and text 714 for indicating that a game item is obtained. When the user interface 710 is displayed, the user emotion determining apparatus may obtain first brain wave data of the user, and determine the obtained first brain wave data as reference biometric information for determining the emotion of the user. For example, when the user interface 710 for providing a game item upon completion of a game quest is displayed as shown in FIG. 7B, the user emotion determining apparatus determines that the user has a positive emotion, and positively obtains the obtained first EEG data. The first reference biometric information for determining the emotion may be determined.

8A and 8B are exemplary diagrams for describing a method for determining reference biometric information in a user emotion determining apparatus according to an embodiment of the present invention. In the following, a user interface including a visual object may be displayed through the display 120 of the HMD device 100 or the display 220 of the image output device 200, and the operations performed below may be performed by the electronic device 300. The processor 340, the processor 240 of the image output device 200, or the processor 150 of the HMD device 100 may be used.

Referring to FIG. 8A, the apparatus for determining a user emotion may display a user interface 800 for solving a problem related to a specific subject. The user interface 800 may include a first visual object 802 related to problem solving. For example, the first visual object 802 may be text representing a problem or / and answer for solving a problem.

The apparatus for determining emotion of a user may acquire first gaze data of the user and determine whether the acquired first gaze data corresponds to the first visual object 802. In various embodiments, the user emotion determining apparatus may obtain first gaze data associated with any one of a plurality of visual objects included in the user interface 800. If the acquired first gaze data corresponds to the first visual object 802, the user emotion determining apparatus may determine whether a specific action of the user with respect to the visual object 802 is detected. For example, a particular action may be the act of choosing an answer to a problem. When the specific behavior is detected, the user emotion determination apparatus may determine whether compensation data is provided according to the result of the specific behavior, and acquire first brain wave data of the user when the compensation data is provided. Specifically, when the selection of the answer to the problem is confirmed, the user emotion determination apparatus checks whether the problem is completed, and when the problem solution is completed, a user interface 810 for providing score data as a reward for completing the problem solution as shown in FIG. 8B. ) Can be displayed. Referring to FIG. 8B, the user interface 810 may include a text / image 812 representing score data acquired according to the completion of the problem solving, a text / image 814 representing the time spent on the problem solving, and a score acquisition. It may include text indicating congratulations (eg, congratulations!) 816. When the user interface 810 is displayed, the user emotion determining apparatus may obtain first brain wave data of the user, and determine the obtained first brain wave data as reference biometric information for determining the emotion of the user. For example, when the user interface 810 for providing the score data according to the completion of the problem solving is displayed as shown in FIG. 8B, the user emotion determining apparatus determines that the user has a positive emotion, and positively obtains the obtained first EEG data. The first reference biometric information for determining the emotion may be determined.

9 is an exemplary diagram for describing a method of generating emotion information indicating a user's emotion with respect to a specific visual object in a user emotion determination apparatus according to an embodiment of the present invention. Operations performed below may be performed through the processor 340 of the electronic device 300, the processor 240 of the image output device 200, or the processor 150 of the HMD device 100.

Referring to FIG. 9, the apparatus for determining user emotion may display a user interface 900 including a second visual object 902 indicating an advertisement target. For example, the second visual object 902 may be a vehicle image.

The apparatus for determining emotion of a user may acquire second gaze data of the user and determine whether the acquired second gaze data corresponds to the second visual object 902. For example, the apparatus for determining a user's emotion obtains second gaze data including position coordinates of a user's gaze, maps a location coordinate of the user's gaze with a specific location coordinate on the display, and thus the mapped coordinates are used to display the vehicle image. It may be determined whether one of the pixels constituting the pixel corresponds to the position coordinate of the pixel. If the mapped coordinates correspond to the position coordinates of one of the pixels constituting the vehicle image, the user emotion determining apparatus may determine that the second gaze data of the user corresponds to the second visual object 902. When the second gaze data of the user corresponds to the second visual object 902, the user emotion determining apparatus obtains the second brain wave data of the user, and based on the reference biometric information and the acquired second brain wave data, the second visual object ( Emotion information representing the emotion of the user with respect to 902 may be generated. For example, the apparatus for determining a user's emotion may identify reference biometric information that matches the brain wave data acquired in response to the visual object 902, and transmit emotion information related to the identified reference biometric information to the second visual object 902. It can be generated as emotion information of the user. When the reference biometric information is associated with a positive emotion, the user emotion determining apparatus may generate the positive emotion as emotion information for the second visual object 902.

Through this, the present invention can check the emotion of the user for the visual object corresponding to the advertisement target even if the user does not express the emotion about the advertisement target. In addition, the present invention is to determine the user's emotion with respect to the second visual object based on the reference biometric information determined using the first visual object that has a feature or theme unrelated to the second visual object or that belongs to a category not related to the second visual object. Can be.

10A and 10B are exemplary diagrams for describing a method for correcting reference biometric information in a user emotion determining apparatus according to an embodiment of the present invention. In the following, a user interface including a visual object may be displayed through the display 120 of the HMD device 100 or the display 220 of the image output device 200, and the operations performed below may be performed by the electronic device 300. The processor 340, the processor 240 of the image output device 200, or the processor 150 of the HMD device 100 may be used.

Referring to FIG. 10A, the apparatus for determining user emotion may display a user interface 1000 including a plurality of

visual objects

1002 and 1004. The user interface 1000 may include a visual object 1002 for a first person and a third visual object 1004 for a second person. In various embodiments, the plurality of visual objects may include images related to various subjects. When the user interface 1000 is displayed, the user emotion determining apparatus may obtain third gaze data of the user and determine whether the acquired third gaze data corresponds to any one of the plurality of

visual objects

1002 and 1004. . For example, when the third gaze data corresponds to the third visual object 1004 among the plurality of

visual objects

1002 and 1004, the user emotion determining apparatus prefers the third visual object 1004 as illustrated in FIG. 10B. A user interface 1010 may be displayed for presentation. According to various embodiments, the user emotion determining apparatus may display a user interface 1010 as illustrated in FIG. 10B when a user input for selecting the third visual object 1004 is received from among the plurality of

visual objects

1002 and 1004. . The user interface 1010 may include an image 1012 and text 1014 to indicate that the third visual object 1004 is preferred.

Referring to FIG. 10B, when the user interface 1010 is displayed, the user emotion determining apparatus may obtain third brain wave data of the user and correct the reference biometric information by using the acquired third brain wave data. For example, the apparatus for determining a user emotion determines that the third brain wave data acquired in response to the third visual object 1004 has a user's preferred emotion, and the third brain wave data and the reference living body related to the preferred emotion. The reference biometric information may be corrected to match the EEG data corresponding to the information.

In this way, the present invention obtains EEG data corresponding to a specific emotion of the user and corrects the reference biometric information by using the acquired EEG data, thereby obtaining a highly accurate determination result of the emotion of the user.

As described above, the present invention generates emotion information about a specific visual object that the user gazes by using the reference biometric information determined based on the gaze data and the brain wave data of the user, thereby visualizing the various visual objects that the user gazes at. Predict a user's feelings for.

Apparatus and method according to an embodiment of the present invention can be implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like.

The program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Hardware devices specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory and the like. In addition, the above-described medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications can be made without departing from the spirit of the present invention. . Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims

Providing a first user interface comprising a first visual object via a Head-Mounted Display (HMD) device configured to obtain first gaze data and first brain wave data of a user;

Acquiring first brain wave data of the user corresponding to the first visual object associated with the first gaze data through the HMD device;

Determining the obtained first brain wave data as reference biometric information for determining the user emotion;

Providing a second user interface comprising a second visual object via the HMD device;

Acquiring second gaze data of the user with respect to the second visual object through the HMD device, and acquiring second brain wave data of the user corresponding to a second visual object associated with the acquired second gaze data; ; And

And generating emotion information on the second visual object based on the reference biometric information and the second brain wave data.
The method of claim 1, wherein the first visual object,

And at least one graphic object related to rewarded content that provides a reward service for a specific action of the user.
The method of claim 2, wherein the obtaining of the first brain wave data of the user comprises:

And acquiring first brain wave data of the user when compensation data is provided as a reward for the specific behavior.
The method of claim 3, wherein the determining of the reference biometric information comprises:

If the reward data is a positive reward, determining the first EEG data as first reference biometric information for determining a positive emotion; And

And determining the first EEG data as second reference biometric information for determining a negative emotion when the compensation data is a negative reward.
The method of claim 1, wherein the second visual object,

At least one graphic object corresponding to a category unrelated to the first visual object.
The method of claim 1,

Providing a third user interface including a third visual object through the user emotion determination device;

Acquiring third brainwave data of the user corresponding to a third visual object associated with the acquired third gaze data when the third gaze data of the user is acquired for the third visual object; And

And correcting the reference biometric information based on the obtained third brain wave data.
In the user emotion determination device,

Providing a first user interface including a first visual object through a head-mounted display (HMD) device configured to acquire a first gaze data and first brain wave data of a user,

Acquire first brain wave data of the user corresponding to the first visual object associated with the first gaze data through the HMD device;

Determining the obtained first brain wave data as reference biometric information for determining the user emotion,

Providing a second user interface including a second visual object through the HMD device;

Obtaining second electroencephalogram data of the user corresponding to the second visual object associated with the acquired second gaze data when the second gaze data of the user is acquired through the HMD device;

And generating emotion information on the second visual object based on the reference biometric information and the second brain wave data.
The method of claim 7, wherein the first visual object,

And at least one graphic object related to rewarded content that provides a reward service for a particular action of the user.
The method of claim 8, wherein the user emotion determination device,

And obtaining first brain wave data of the user when reward data is provided as a reward for the specific behavior.
The method of claim 9, wherein the user emotion determination device,

If the reward data is a positive reward, the first EEG data is determined as first reference biometric information for determining a positive emotion,

And determining the first EEG data as second reference biometric information for determining negative emotion when the compensation data is negative compensation.
The method of claim 7, wherein the second visual object,

And at least one graphic object corresponding to a category unrelated to the first visual object.
The method of claim 7, wherein the user emotion determination device,

Providing a third user interface including a third visual object,

Obtaining third brain wave data of the user corresponding to a third visual object associated with the acquired third eye data when the third eye data of the user is acquired for the third visual object,

And correcting the reference biometric information based on the obtained third brain wave data.