KR20220103382A - Electronic device identifying emotion state and operating method thereof - Google Patents

Abstract

An electronic device comprises: a display; at least one sensor; a processor; and a memory operatively coupled to the display, the at least one sensor, and the processor, and storing instructions. When the instructions are executed by the processor, the electronic device may be set to run an application providing at least one piece of content provided to the electronic device and at least one external electronic device, display the at least one piece of content through the display, specify a portion of the at least one piece of content by using the at least one sensor, identify the emotional state of a user with respect to the specified content by using the at least one sensor, and display a visual object in association with the specified content on the basis of the identified emotional state. Various other embodiments identified through the specification are possible. According to the present invention, the electronic device is capable of reflecting a user's emotional reaction to the content in real time.

Description

Electronic device for identifying emotional state and operating method thereof

The present disclosure relates to an electronic device for identifying an emotional state and an operating method thereof.

The user may emotionally respond to content (eg, a message, image, video) provided by the electronic device. For example, if the content provided to the user includes a fun element, the user may check the content and show emotion (eg, laughter) related to the content.

A user may display an emotional state for the content. For example, the user may tag an emoticon or an animoji indicating his/her emotion with respect to specific content.

In order to display the user's emotional state for the content displayed on the electronic device, a user's input (eg, tagging of an emoticon or an Animoji) for the content may be required. In this case, it may be difficult for the electronic device to reflect the user's emotional state for the content in real time.

An electronic device according to an embodiment disclosed in this document includes a display, at least one sensor, a processor, and a memory that is operatively connected to the display, the at least one sensor, and the processor, and stores instructions, , when the instructions are executed by the processor, the electronic device executes an application that provides at least one content to the electronic device and at least one external electronic device, and displays the at least one content through the display. display, specify a part of the at least one content using the at least one sensor, identify the user's emotional state for the specified content using the at least one sensor, and respond to the identified emotional state based on the specified content and may be configured to display a visual object.

In addition, the method of operating an electronic device according to an embodiment disclosed in this document includes an operation of executing an application that provides at least one content to the electronic device and at least one external electronic device, and displaying the at least one content operation, an operation of specifying a part of the at least one content, an operation of identifying a user's emotional state with respect to the specified content, and displaying a visual object in association with the specified content based on the identified emotional state It may include an action to

According to the embodiments disclosed in this document, the electronic device may reflect the user's emotional response to the content in real time.

According to the embodiments disclosed in this document, the electronic device may reflect the accumulated user's and external user's emotional responses to the content step by step.

In addition, various effects directly or indirectly identified through this document may be provided.

Effects that can be obtained in various embodiments of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned are clearly to those of ordinary skill in the art to which the present disclosure belongs from the description below. can be understood

1 is a block diagram of an electronic device in a network environment, according to various embodiments of the present disclosure;
2 illustrates a configuration of an electronic device according to an embodiment of the present disclosure.
3 illustrates an execution screen of an application of an electronic device according to an embodiment of the present disclosure.
4A is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.
4B is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.
5 illustrates a visual object update according to an embodiment of the present disclosure.
6 illustrates user settings according to an embodiment of the present disclosure.
7 is a diagram illustrating story generation according to an embodiment of the present disclosure.
8 illustrates a visual object according to various embodiments of the present disclosure.
9 illustrates an AR device according to an embodiment of the present disclosure.
10 illustrates augmented reality, according to an embodiment of the present disclosure.
11A illustrates augmented reality according to an embodiment of the present disclosure.
11B illustrates augmented reality according to an embodiment of the present disclosure.
12 illustrates a holocall function according to an embodiment of the present disclosure.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments of the present disclosure. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 . In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 120 stores a command or data received from another component (eg, the sensor module 176 or the communication module 190 ) into the volatile memory 132 . may be stored in , process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 uses less power than the main processor 121 or is set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The secondary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the auxiliary processor 123 (eg, image signal processor or communication processor) may be implemented as a part of another functionally related component (eg, camera module 180 or communication module 190). have. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which artificial intelligence is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . The electronic device 102) (eg, a speaker or headphones) may output a sound.

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various techniques for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements defined in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 includes a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: Downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal (eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of the operations executed by the electronic device 101 may be executed by one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. The server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

The electronic device according to various embodiments disclosed in this document may be a device of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C," each of which may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

According to various embodiments of the present document, one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101) may be implemented as software (eg, the program 140) including For example, the processor (eg, the processor 120 ) of the device (eg, the electronic device 101 ) may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to an embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store™) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. . According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

2 illustrates a configuration of an electronic device according to an embodiment of the present disclosure. 3 illustrates an execution screen of an application of an electronic device according to an embodiment of the present disclosure.

According to an embodiment, the electronic device 200 (eg, the electronic device 101 of FIG. 1 ) includes a processor 210 , a display 220 , at least one sensor 230 , and/or a memory 240 . may include The electronic device 200 shown in FIG. 2 may further include at least some of the components of the electronic device 101 of FIG. 1 . For example, the electronic device 200 may further include the battery 189 of FIG. 1 .

According to one embodiment, the processor 210 , the display 220 , the at least one sensor 230 , or the memory 240 includes the processor 120 , the display module 160 , the sensor module 176 of FIG. 1 , Alternatively, each may correspond to the memory 130 .

According to an embodiment, the processor 210 may execute an application (eg, the application 146 of FIG. 1 ). According to an embodiment, the application may be an application for providing at least one content. For example, the application may include a messenger, and/or social media (eg, a social network service).

According to an embodiment, the processor 210 may display at least one content provided through an application through the display 220 . For example, the at least one content may include a text message, an Internet post, an image, and/or a video.

According to an embodiment, the execution screens 300a, 300b, and 300c of FIG. 3 may be screens generated by an application. According to an embodiment, the execution screens 300a, 300b, and 300c of FIG. 3 may represent a group communication screen in which a plurality of users participate.

Referring to the execution screen 300a of FIG. 3 , the processor 210 may display at least one content 310 , 315 through the display 220 . The at least one content 310 or 315 may include a text message, an image, and/or an image.

According to an embodiment, the processor 210 may identify the user's gaze point (or eye tracking) using the at least one sensor 230 . The processor 210 may obtain the user's right eye image and the user's left eye image using at least one sensor 230 , respectively, and identify the user's gaze information based on the user's right eye image and the left eye image. The processor 210 may identify the user's gaze point based on the gaze information.

According to an embodiment, the processor 210 may specify a partial area (or content) that the user is looking at from among at least one content based on the user's gaze point.

Referring to FIG. 3 , the processor 210 may specify the content 310 corresponding to the user's gaze point among at least one content 310 and 315 displayed through the execution screen 300a.

According to an embodiment, the processor 210 may obtain a face image of the user by using at least one sensor 230 , and may obtain expression information of the user based on the face image. The processor 210 identifies the shape of the user's landmark (eg, eyes, eyebrows, lips, nose) in the face image, and acquires facial expression information (eg, smile) of the user based on the shape of the user's landmark. can

According to an embodiment, the processor 210 may identify the emotional state of the user based on the facial expression information. The emotional state may include joy, sadness, anger, happiness, disgust, and/or surprise. According to an embodiment, the processor 210 may identify that the user is in a happy state based on the user's smile.

According to an embodiment, the processor 210 may identify the user's emotional state indicated while the user gazes at the content 310 .

According to an embodiment, the processor 210 may display a visual object in association with the content 310 in which the emotional state is identified. For example, the processor 210 may change the display of the content 310 in which the emotional state is identified, or the peripheral region 320 of the content 310 . According to an embodiment, the visual object may include, for example, a color, an icon, an emoticon, and/or an avatar.

Referring to the execution screen 300a of FIG. 3 , the processor 210 may display a visual object in the peripheral area 320 of the content 310 in which the emotional state is identified. For example, the processor 210 may change the color of the peripheral region 320 of the content 310 to a color (eg, sky blue) that is distinguished from other regions of the display 220 .

According to an embodiment, the processor 210 may transmit the user's emotional state information identified with respect to the content 310 to at least one server (eg, the server 108 of FIG. 1 ). The emotional state information may include information about the identified emotion type (eg, joy, sadness, anger, happiness, disgust, and/or surprise) and target content (eg, content 310 ). When there is a request from at least one external electronic device (eg, the electronic device 101 of FIG. 1 ), the at least one server transmits the emotional state information received from the electronic device 200 to the at least one external electronic device. can

According to an embodiment, the processor 210 may receive the emotional state information of the external user for the content 310 from at least one server. According to an embodiment, at least one external electronic device (eg, the electronic device 101 of FIG. 1 ) executes an application to externally transmit at least one content 310 , 315 identical to that provided by the electronic device 200 . can be provided to users. The at least one external electronic device 101 may identify the emotional state of the external user with respect to the at least one content 310 , 315 . For example, when the external user's emotional state with respect to the content 310 is identified, at least one external electronic device may transmit the external user's emotional state information to at least one server. The emotional state information may include information about the identified emotion type (eg, joy, sadness, anger, happiness, disgust, and/or surprise) and target content (eg, content 310 ). According to an embodiment, the processor 210 may acquire the number of times of identification of the external user's emotional state with respect to the target content (eg, the content 310 ) based on the emotional state information received from at least one server.

According to an embodiment, the processor 210 may update the execution screen 300a like the execution screen 300b based on the received emotional state information. For example, when the emotional state of one or more external users is identified with respect to the content 310 , the processor 210 may change the color of the peripheral area 320 of the content 310 to a color with a deeper saturation (eg, blue). ) can be changed to

According to an embodiment, the processor 210 may update the visual object based on the number of times of identification of the external user's emotional state with respect to the content 310 . For example, the processor 210 may change the color of the surrounding area 320 of the content 310 to a color (eg, navy blue) with a deeper saturation than before (eg, blue) as the number of times of identification of the emotional state increases. have.

According to an embodiment, the processor 210 may update the execution screen 300b like the execution screen 300c by adding a visual object to the peripheral area 320 of the content 310 whose emotional state is identified. . For example, the processor 210 may display the avatar 330 in the peripheral area 320 of the content 310 . The avatar 330 may correspond to at least some of the plurality of users participating in the group chat. Each of the avatars 330 may represent an emotional state for the corresponding user's content 310 . According to an embodiment, at least some users' emotional states for the content 310 may be obtained directly or indirectly (or through at least one server) from an external electronic device (eg, at least some users' electronic devices). can

4A is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure. 4B is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 4A , in operation 400 , a processor (eg, the processor 210 of FIG. 2 ) may execute an application. The application may include, for example, a messenger, and/or social media (eg, a social network service). The application may also be executed by at least one external electronic device (eg, the electronic device 101 of FIG. 1 ).

In operation 410 , the processor 210 may display at least one content through a display (eg, the display 220 of FIG. 2 ). At least one content may be provided to the electronic device 200 through an application. The at least one content may include a text message, an image, and/or an image. Also, at least one content may be provided to at least one external electronic device through an application.

In operation 420 , the processor 210 may specify a part of at least one piece of content. The processor 210 may identify the user's gaze point using the at least one sensor 230 and specify a part of at least one piece of content corresponding to the user's gaze point.

In operation 430, the processor 210 may identify the user's emotional state. The processor 210 may obtain facial expression information of the user using at least one sensor 230 . The processor 210 may identify the user's emotional state based on facial expression information (eg, a smile). The emotional state may include, for example, joy, sadness, anger, happiness, disgust, and/or surprise. The emotional state in operation 430 may be an emotional state for some of the at least one content specified in operation 420 .

In operation 440 , the processor 210 may display a visual object in association with a portion of at least one content. According to an embodiment, the processor 210 may display the content specified in operation 420 and the visual object in a region surrounding the content. Visual objects may include, for example, colors, icons, emoticons, and/or avatars. The description of the visual object is exemplary, and the type of the visual object is not limited by the embodiment of the present document.

Referring to FIG. 4B , descriptions of operations 400 and 410 may be referred to with reference to the description of FIG. 4A . In the description of FIG. 4B , a description of the operations of FIG. 4A may be referred to by reference to the description of FIG. 4A .

In operation 450 , the processor 210 may receive the emotional state information of the external user identified by the at least one external electronic device from at least one server (eg, the server 108 of FIG. 1 ). According to an embodiment, at least one content may be provided to at least one external electronic device through an application. The at least one external electronic device may identify the emotional state of the external user with respect to some of the at least one content (eg, the content specified in operation 420 ). At least one external electronic device may transmit the identified emotional state information to at least one server. The emotional state information that the processor 210 receives from at least one server includes the identified emotion type (eg, joy, sadness, anger, happiness, disgust, and/or surprise) and target content (eg, content 310 ). may contain information.

In operation 460, the processor 210 may update the visual object based on the emotional state information. According to an embodiment, the processor 210 may update the displayed visual object in association with some of the at least one content (eg, the content specified in operation 420 ) based on the emotional state information received in operation 450 . For example, when the emotional state of at least one or more external users is identified with respect to a part of at least one piece of content, the processor 210 may update the displayed visual object in operation 440 . For example, when the visual object is a color (eg, sky blue), the processor 210 may display a more saturated color (eg, sky blue) in a region surrounding the content specified in operation 420 .

According to an embodiment, when the emotional state of the user of the electronic device 200 is not identified in operation 430 , the processor 210 may display a visual object based on the emotional state information received in operation 450 .

The processor 210 may reflect the emotional state of the user and the external user with respect to specific content on the application execution screen in real time through the operations of FIGS. 4A and 4B . Also, the processor 210 may update the visual object based on the number of times of identification of the external user's emotional state with respect to the content 310 . For example, the processor 210 may change the color of the surrounding area 320 of the content 310 to a color (eg, navy blue) with a deeper saturation than before (eg, blue) as the number of times of identification of the emotional state increases. have. The processor 210 may indicate accumulated emotions of a plurality of users by differentiating the display of the visual object.

5 illustrates a visual object update according to an embodiment of the present disclosure.

According to an exemplary embodiment, the at least one external electronic device 510 (eg, the electronic device 101 of FIG. 1 ) is a cloud 520 (eg, the server 108 of FIG. 1 ), which is the at least one It is possible to transmit emotional state information about the content. The processor (eg, the processor 210 of FIG. 2 ) of the electronic device (eg, the electronic device 200 of FIG. 2 ) may receive the emotional state information of the external user from the cloud 520 . The emotional state information may include information about the identified emotion type (eg, joy, sadness, anger, happiness, disgust, and/or surprise) and target content (eg, content 310 ). FIG. 5 may update a visual object associated with at least one content based on the received emotional state information. P1 may correspond to the electronic device 200 . P2 , P3 , and P4 may correspond to each of the at least one external electronic device 510 .

According to an embodiment, reference numerals 530, 540, and 550 denote the electronic device 200 and at least one external electronic device 510 for a specific time point (eg, 14:00, 15:23, 16:14). Shows whether a specific content (eg, a part of at least one content) is checked and an emotional state is identified. Reference numerals 532, 542, and 552 show confirmation states of users and external users for specific content. The electronic device 200 and the at least one external electronic device 510 may determine whether to check the content based on gaze information of the user and the external user. For example, the electronic device 200 and the at least one external electronic device 510 may identify the user's gaze point based on gaze information. When the user's gaze point corresponds to the specific content, the electronic device 200 and the at least one external electronic device 510 may identify that the user has checked the specific content. According to an embodiment, the processor 210 may display whether the user and the external user have checked the specific content through a user interface (UI). For example, the processor 210 may display the number of external users who have not checked the content on the group communication screen around the specific content.

According to an embodiment, reference numerals 534, 544, and 554 indicate whether the emotional state of the user and the external user for specific content is identified. According to an embodiment, the processor 210 may identify a user's emotional state for specific content using at least one sensor (eg, at least one sensor 230 ). According to an embodiment, the processor 210 may receive the emotional state information of the external user from the cloud 520 to identify whether the external user's emotional state for specific content is identified.

According to an embodiment, at reference numeral 530 , it may be understood that the user of the electronic device 200 identifies specific content and the processor 210 identifies the user's emotional state. The processor 210 may display a visual object in association with specific content based on the user's emotional state. For example, the processor 210 may display a color (eg, sky blue) that is distinguished from other areas of the display 220 in the area surrounding the specific content.

According to an embodiment, at reference number 530 , the processor 210 may identify whether the external user has checked specific content based on the emotional state information received from the cloud 520 . The processor 210 checks the specific content by the external user of the external electronic device P2 based on the received emotional state information, but the external electronic device P2 determines the external user's emotional (eg, joy) state for the specific content. can be determined as not being identified. The processor 210 may determine that the external user of the external electronic devices P3 and P4 has not checked the specific content based on the received emotional state information.

According to an embodiment, at reference numeral 540 , the processor 210 may additionally receive the emotional state information of the external user from the cloud 520 . The processor 210 identifies the specific content by the external user of the external electronic device P3 based on the received emotional state information, and the external electronic device P2 determines the external user's emotional (eg, laughter) state for the specific content. can be determined to be identified. The processor 210 may update the visual object associated with the specific content based on the emotional state of the external user. For example, the processor 210 may display a color (eg, blue) with a deeper saturation than the color (eg, sky blue) at the time point 530 in the peripheral area of the specific content. The electronic device 200 may determine that the external user of the external electronic device P4 has not checked the specific content based on the received emotional state information.

According to an embodiment, at reference numeral 550 , the processor 210 may additionally receive the emotional state information of the external user from the cloud 520 . The processor 210 checks the specific content by the external user of the external electronic device P4 based on the received emotional state information, and the external electronic device P4 determines the external user's emotional (eg, joy) state for the specific content. can be determined to be identified. The processor 210 may update a visual object associated with specific content. For example, the processor 210 may display a color (eg, indigo blue) with a deeper saturation than the color (eg, blue) displayed at the time point of reference number 540 in the peripheral area of the specific content.

5 , the processor 210 may update the visual object in real time based on whether the external user confirms the specific content and whether the emotional state is identified. The user may continuously monitor the emotional response of the user and the external user to the specific content through the visual object.

6 illustrates user settings according to an embodiment of the present disclosure.

According to an embodiment, the processor 210 may provide a graphic object 610 for user setting related to emotional state information on the application execution screen 600a. The processor 210 may display the execution screen 600b in response to a user input for the graphic object 610 . The execution screen 600a may correspond to the execution screen 300a of FIG. 3 .

According to an embodiment, the processor 210 may display the graphic object 620 on the execution screen 600b. The graphic object 620 may be a user interface (UI) for setting whether to share the emotional state of the user identified by the electronic device 200 . For example, the processor 210 may not receive the user input for the graphic object 620 and transmit the identified emotional state of the user to the cloud (eg, the cloud 520 of FIG. 5 ). As another example, the processor 210 may receive a user input for the graphic object 620 and transmit the identified emotional state of the user to the cloud (eg, the cloud 520 of FIG. 5 ).

7 is a diagram illustrating story generation according to an embodiment of the present disclosure.

In the execution screen 700a, the processor (eg, the processor 210 of FIG. 2 ) may provide the content 705 through a display (eg, the display 220 of FIG. 2 ). Content 705 may be a still image and/or video. When the user's emotional state (eg, joy) for the content 705 is identified, the processor 210 may display the emoticon 710 in association with the content 705 as in the execution screen 700b. . The content 705 in which the emoticon 710 is displayed may be understood as content in which the user's emotional state is identified.

According to an embodiment, the processor 210 may generate a story by gathering at least one content in which the user's emotional state is identified. The processor 210 may provide the generated story as an execution screen 700c. The processor 210 may display the user's emotional state identified with respect to at least one content included in the story as the story title 720 .

8 illustrates a visual object according to various embodiments of the present disclosure.

The processor 210 may display an execution screen 800a on a display (eg, the display 220 of FIG. 2 ) by executing an application (eg, social media). The processor 210 may display at least one content 802 , 804 , and 806 through the execution screen 800a . The at least one content 802 , 804 , and 806 may be provided to at least one external electronic device (eg, the electronic device 101 of FIG. 1 ) through an application. The processor 210 may receive the emotional state information of the external user from the cloud (eg, the cloud 520 of FIG. 5 ) and update the visual objects 810 and 820 . For example, the visual object (eg, 810, 820) may indicate the number of times of identification of the external user's emotional state with respect to the corresponding content (eg, 802, 804).

The processor 210 may display an execution screen 800b on the display 220 by executing an application (eg, the Internet). The processor 210 displays a color (eg, sky blue) that is distinguished from other areas of the display 220 in the peripheral area 835 of the content 830 based on the emotional state of the user and the external user for the content 830 . can do. Content 830 may be an Internet post.

9 illustrates an AR device according to an embodiment of the present disclosure.

According to an embodiment, the electronic device of FIG. 2 (eg, the electronic device 200 of FIG. 2 ) may be an AR device that provides augmented reality (AR). 9 is a schematic diagram of an electronic device 901 (eg, the electronic device 200 of FIG. 2 ) when the electronic device 200 is an AR device.

Referring to FIG. 9 , in the example of FIG. 9 , the electronic device 901 may be referred to as a head mounted display (HMD) device, a wearable device, smart glasses, or eyewear. The shape of the electronic device 901 illustrated in FIG. 9 is exemplary, and embodiments of the present document are not limited thereto. For example, the electronic device 901 may be any electronic device configured to provide augmented reality (AR) or virtual reality (VR).

According to an embodiment, the electronic device 901 may include at least some of the components of the electronic device 101 of FIG. 1 . For example, the electronic device 901 includes a display (eg, the display module 160 of FIG. 1 ), a camera (eg, the camera module 180 of FIG. 1 ), and at least one sensor (eg, the sensor module of FIG. 1 ). 176 ), a processor (eg, processor 120 in FIG. 1 ), a battery (eg, battery 189 in FIG. 1 ), memory (eg, 130 in FIG. 1 )), or communication circuitry (eg, FIG. 1 ). of the communication module 190). At least some of the components of the electronic device 901 may be located inside the housing of the electronic device 901 or may be exposed to the outside of the housing.

The electronic device 901 may include a display (eg, the display 220 of FIG. 2 ). For example, the electronic device 901 may include a first display 961-1 and/or a second display 961-2. The first display 961-1 and/or the second display 961-2 may include a liquid crystal display (LCD), a digital mirror device (DMD), and a silicon liquid crystal display (liquid crystal). It may include at least one of an on silicon device (LCoS device), an organic light emitting diode (OLED), and a micro light emitting diode (micro LED).

According to an embodiment, the electronic device 901 may include a first transparent member 996-1 and/or a second transparent member 996-2. For example, when the user wears the electronic device 901 , the user may see through the first transparent member 996 - 1 and/or the second transparent member 996 - 2 . The first transparent member 996-1 and/or the second transparent member 996-2 may be formed of at least one of a glass plate, a plastic plate, or a polymer, and may be transparent or translucent. For example, when worn, the first transparent member 996-1 may be disposed to face the user's right eye, and the second transparent member 996-2 may be disposed to face the user's left eye.

According to an embodiment, at least a portion of the first transparent member 996-1 and/or the second transparent member 996-2 may be an optical waveguide. For example, the optical waveguide may transmit an image generated by a display (eg, the first display 961-1 and/or the second display 961-2) to the user's eyes. The optical waveguide may be formed of glass, plastic, or polymer.

9 illustrates that the electronic device 901 provides an image to a user using an optical waveguide, but embodiments of the present document are not limited thereto. According to an embodiment, the display of the electronic device 901 may be a transparent or semi-transparent display. In this case, the display may be disposed at a position facing the user's eyes (eg, the first screen output area 960-1 and/or the second screen output area 960-2).

According to an embodiment, the electronic device 901 may include at least one camera. For example, the electronic device 901 may include a first camera 980-1, a second camera 980-2, and/or a third camera 980-3. For example, the first camera 980-1 and the second camera 980-2 may be used for external image recognition. The first camera 980-1 and the second camera 980-2 may be set to acquire an image corresponding to a direction (eg, a +x direction) corresponding to the user's gaze. The electronic device 901 uses a first camera 980-1 and a second camera 980-2 to perform head tracking (eg, 3 degrees of freedom or 6 degrees of freedom (DoF) tracking). ), hand image detection, hand image tracking, and/or spatial recognition. For example, the first camera 980-1 and the second camera 980-2 have the same standard and performance (eg, angle of view, shutter speed, resolution, and/or number of color bits, etc.) shutter) may be a camera. The electronic device 901 implements a simultaneous localization and mapping (SLAM) technology by performing spatial recognition (eg, 6-DOF spatial recognition) and/or depth information acquisition using stereo cameras disposed on the left and right sides. can support Also, the electronic device 901 may recognize a user's gesture using stereo cameras disposed on the left/right side. The electronic device 901 may detect a faster hand gesture and fine movement by using a GS camera having relatively less distortion than a rolling shutter (RS) camera. For example, the third camera 980 - 3 may be used for external image recognition. The third camera 980 - 3 may be set to acquire an image corresponding to a direction (eg, a +x direction) corresponding to the user's gaze. In one example, the third camera 980 - 3 may be a camera having a relatively higher resolution than that of the first camera 98 - 1 and the second camera 980 - 2 . The third camera 980 - 3 may be referred to as a high resolution (HR) camera or a photo video (PV) camera. The third camera 980 - 3 may support functions for acquiring high-quality images, such as auto focus (AF) and/or optical image stabilization (OIS). The third camera 980 - 3 may be a GS camera or an RS camera.

According to an embodiment, the electronic device 901 may include at least one eye-tracking sensor. For example, the electronic device 901 may include a first eye tracking sensor 976 - 1 and a second eye tracking sensor 976 - 2 . The first eye tracking sensor 976 - 1 and the second eye tracking sensor 976 - 2 may be, for example, cameras set to acquire an image in a direction corresponding to the user's eyes. The first eye tracking sensor 976 - 1 and the second eye tracking sensor 976 - 2 may be configured to respectively acquire the user's right eye image and the user's left eye image. The electronic device 901 may be configured to detect the user's pupil using the first eye tracking sensor 976 - 1 and the second eye tracking sensor 976 - 2 . The electronic device 901 may obtain the user's gaze from the user's pupil image, and provide the image based on the obtained gaze. For example, the electronic device 901 may display the image so that the image is located in the user's gaze direction. For example, the first eye tracking sensor 976-1 and the second eye tracking sensor 976-2 may have the same specifications and performance (eg, angle of view, shutter speed, resolution, and/or number of color bits). It may be a GS (global shutter) camera with

According to an embodiment, the electronic device 901 may include at least one printed circuit board (PCB). For example, the electronic device 901 includes a first PCB 987-1 positioned in a first temple 998-1 and a second PCB 987-2 positioned in a second temple 998-2. ) may be included. The first PCB 987 - 1 and/or the second PCB 987 - 2 may be electrically connected to other components of the electronic device 901 through a signal line and/or a flexible PCB (FPCB). For example, communication circuitry, memory, at least one sensor, and/or a processor may be disposed on the first PCB 987 - 1 and/or the second PCB 987 - 2 . For example, each of the first PCB 987 - 1 and the second PCB 987 - 2 may be composed of a plurality of PCBs spaced apart by an interposer.

According to an embodiment, the electronic device 901 may include a first temple 998 - 1 , a second temple 998 - 2 , and a frame 997 . The first temple 998-1, the second temple 998-2, and the frame 997 may be referred to as a housing. The first temple 998 - 1 is physically connected to the frame 997 through the first hinge portion 999 - 1 , and may support the frame 997 when worn. The second temple 998 - 2 is physically connected to the frame 997 through the second hinge portion 999 - 2 , and may support the frame 997 when worn.

The configuration of the above-described electronic device 901 is exemplary, and embodiments of the present document are not limited thereto. For example, the electronic device 901 may not include at least some of the components described with reference to FIG. 9 , or may further include components other than the components described with reference to FIG. 9 . For example, the electronic device 901 may include at least one sensor (eg, an acceleration sensor, a gyro sensor, and/or a touch sensor) and/or an antenna.

10 illustrates augmented reality, according to an embodiment of the present disclosure.

The electronic device (eg, the electronic device 901 of FIG. 9 ) may be worn on a part of the user's body (eg, the head). The electronic device 901 may include a display capable of supporting augmented reality (eg, the first display 961-1 and/or the second display 961-2 of FIG. 9 ). The electronic device 901 may provide the augmented reality 10 through a display.

According to an embodiment, the processor of the electronic device 901 (eg, the processor 210 of FIG. 2 ) may execute an application. For example, the application may provide a group communication function.

According to an embodiment, the processor 210 may output a graphic user interface (GUI) 1030 in the augmented reality 10 . The GUI 1030 may correspond to one group communication. When the processor 210 executes a plurality of group communications on an application, the processor 210 may output a plurality of GUIs in the augmented reality 10 . The GUI 1030 may include avatars 1010 , 1012 , and 1014 corresponding to users participating in group communication and external users. For example, the avatar 1014 may correspond to a user of the electronic device 901 . The avatars 1010 and 1012 may correspond to external users participating in group communication. Although the GUI 1030 is described herein as 'comprising' an avatar, according to various embodiments, the avatars 1010 , 1012 , and 1014 are disposed in various locations adjacent to the GUI 1030 on the augmented reality 10 . can be For example, avatars 1010 , 1012 , 1014 may be placed on top of GUI 1030 .

According to an embodiment, the processor 210 may output at least one content 1005 transmitted or received by the user and the external user through group communication in association with the avatars 1010 , 1012 , and 1014 . The at least one content 1005 may include a text message, an image, and/or a video.

According to an embodiment, the processor 210 collects the user's gaze information using a gaze tracking sensor (eg, the first gaze tracking sensor 976-1 and the second gaze tracking sensor 976-2 of FIG. 9). acquired, and the user's gaze point may be identified based on the acquired gaze information. Referring to reference numeral 1000b, the processor 210 may specify the content 1040 corresponding to the user's gaze point.

According to an embodiment, the processor 210 obtains facial expression information of the user for the content 1040 using at least one sensor (eg, the at least one sensor 230 of FIG. 2 ), and based on the facial expression information to identify the user's emotional state (eg, joy, sadness, anger, happiness, disgust, and/or surprise).

According to an embodiment, the processor 210 may receive the emotional state information of the external user with respect to the content 1040 from at least one server (eg, the server 108 of FIG. 1 ).

The processor 210 may display the visual object in association with the content 1040 or update the displayed visual object based on the emotional state of the user identified by the electronic device 901 and/or the received emotional state information of the external user. have. For example, the processor 210 may output an effect (eg, highlight) different from other content on the periphery of the content 1040 . According to an embodiment, as the number of times of identification of the external user's emotional state with respect to the content 1040 increases, the processor 210 may further emphasize the visual object displayed in association with the content 1040 . For example, the processor 210 may increase the intensity or size of an effect (eg, highlight) output to the periphery of the content 1040 .

11A illustrates augmented reality according to an embodiment of the present disclosure. 11B illustrates augmented reality according to an embodiment of the present disclosure.

Referring to FIG. 11A , an electronic device (eg, the electronic device 901 of FIG. 9 ) may be worn by a user 1105 . At reference numeral 1100a, the electronic device 901 may output a plurality of GUIs 1110, 1112, and 1114 to the augmented reality 10 by executing an application. The application may provide a group communication function. The plurality of GUIs 1110 , 1112 , and 1114 may correspond to each of a plurality of group communications executed by an application. The plurality of GUIs 1110 , 1112 , and 1114 may include at least one avatar participating in a corresponding group communication.

According to an embodiment, the processor 210 displays a visual object (eg : highlight) can be displayed. The higher the number of identifications of the emotional state in the group communication, the more highlighted the visual object (eg, highlight) can be displayed. For example, the GUI 1110 may display a stronger visual object (eg, highlight) than the other GUIs 1112 and 1114 . In this case, it may be understood that the number of emotional state identification in group communication corresponding to the GUI 1110 is relatively higher than the number of emotional state identification in group communication corresponding to other GUIs 1112 and 1114 .

The processor 210 uses a camera (eg, the first camera 980-1, the second camera 980-2, and/or the third camera 980-3 of FIG. 9) to the user's gaze 1120 ) can be traced. The processor 210 may identify the user's gaze point by tracking the user's gaze. When the user's gaze point corresponds to the GUI 1110 , the processor 210 may enlarge the specified GUI 1110 as indicated by reference numeral 1100b.

At reference numeral 1100b, the processor 210 may output at least one content 1130 generated in group communication corresponding to the GUI 1110 in association with at least one avatar 1150 and 1155 .

According to an embodiment, a visual object may be displayed in some (eg, content 1140 ) of at least one content 1130 generated in group communication. The description of the display of the visual object may be referred to by the description of FIG. 10 .

Referring to FIG. 11B , at reference numeral 1100c , the processor 210 may identify the gaze point by tracking the gaze 1125 of the user. According to an embodiment, the user's gaze point may correspond to a part of the at least one content 1130 (eg, the content 1140 ). The content 1140 may be content in which a user's emotional state is identified.

At reference numeral 1100d, the processor 210 may project the emotional state identified for the content 1140 onto at least one avatar 1150 and 1155 . For example, when a user's emotion (eg, joy) is identified with respect to the content 1140 , the processor 210 may control the emotion (eg, joy) identified in the facial expressions and actions of the at least one avatar 1150 and 1155 . may be projected and output as at least one avatar 1152 and 1157 .

11 , the processor 210 may specify a part of a plurality of group communication and at least one content by tracking the user's gaze, and reproduce the emotional state identified for the specified content through at least one avatar. have.

12 illustrates a holocall function according to an embodiment of the present disclosure.

According to an embodiment, the electronic device (eg, the electronic device 901 ) may provide a holocall function in the augmented reality 10 . The holo-call function may be a call function performed in the augmented reality 10 . The holo-call function may output a call party as an avatar in the augmented reality 10 . The processor 210 may project the other party's voice, emotion, and/or behavior to the avatar when performing the holo-call function. The processor 210 may receive the emotional state information of the counterpart from the counterpart's electronic device (eg, the electronic device 101 of FIG. 1 ) and/or at least one server (eg, the server 108 of FIG. 1 ). . The emotional state information may include information on the identified emotion type (eg, joy, sadness, anger, happiness, disgust, and/or surprise) and/or behavior (or gesture) information of the other party.

According to an embodiment, the electronic device 901 may store the content of the call. The processor 210 may receive a user input and output a user interface (UI) 1200 for playing the stored call contents to the augmented reality 10 .

According to an embodiment, the UI 1200 may include a graphic object 1205 for reproduction and graphic objects 1207 , 1210 , 1220 , and 1230 indicating a specific section of call contents (eg, voice). The processor 210 may receive a user input for the graphic object 1205 for reproduction and reproduce the content of the call. The processor 210 may receive a user input for a play progress bar 1207 and play call contents corresponding to a location desired by the user.

According to an embodiment, the processor 210 may highlight and display a section in which the emotional state of the other party is identified. For example, it may be understood that the emotional state of the other party is identified in the sections 1210 , 1220 , and 1230 of FIG. 12 . The processor 210 may receive the user input for the sections 1210 , 1220 , and 1230 , and output the avatars 1215 , 1225 , and 1235 corresponding to the counterpart while playing the call contents of the corresponding section. The processor 210 may project the emotional state of the counterpart identified in the corresponding section on the avatars 1215 , 1225 , and 1235 . For example, the processor 210 may receive a user input for the section 1210 and output the avatar 1215 in which the emotional state of the other party identified in the section 1210 is projected.

In FIG. 12 , the user input may include a user's gesture input for a visual object output from the augmented reality 10 . The processor 210 may identify the user's gesture using at least one sensor (eg, at least one sensor 230 of FIG. 2 ). The user's gesture may include performing a specified action by the user's body part (eg, hands and feet) or gazing at a point identified based on the user's gaze tracking. The description of the user input of the present disclosure is illustrative, and the embodiment of the present document is not limited thereto.

Claims (20)

In an electronic device,
display;
at least one sensor;
processor; and
a memory operatively connected to the display, the at least one sensor, and the processor and storing instructions;
The instructions, when executed by the processor, cause the electronic device to:
executing an application that provides at least one content to the electronic device and at least one external electronic device;
display the at least one content through the display,
specifying a part of the at least one content using the at least one sensor,
identifying the user's emotional state for the specified content using the at least one sensor,
configured to display a visual object in association with the specified content based on the identified emotional state;
electronic device.
The method of claim 1,
comprising a wireless communication circuit;
The instructions, when executed by the processor, cause the electronic device to:
receiving emotional state information identified by the at least one external electronic device with respect to the at least one content through the wireless communication circuit;
configured to update the visual object based on the received emotional state information,
electronic device.
The method of claim 1,
The instructions, when executed by the processor, cause the electronic device to:
Obtaining the user's gaze information using the at least one sensor,
configured to specify a portion of the at least one content based on the gaze information,
electronic device.
3. The method of claim 2,
The instructions, when executed by the processor, cause the electronic device to:
Provides a user interface (UI) related to the sharing of the emotional state information,
configured to determine whether to receive or transmit the emotional state information based on a user input to the UI,
electronic device.
3. The method of claim 2,
The instructions, when executed by the processor, cause the electronic device to:
The emotional state information is configured to collect the identified content (gather) to create a story (story) content,
electronic device.
3. The method of claim 2,
The application provides a group communication or SNS (social network service) function,
electronic device.
The method of claim 1,
The visual object includes a color, an icon, an emoticon, or an avatar on which the emotional information is projected,
electronic device.
3. The method of claim 2,
The electronic device is an AR device that provides augmented reality (AR),
The instructions, when executed by the processor, cause the electronic device to:
outputting at least one avatar related to the application and the at least one content in the augmented reality,
and update the visual object based on the received emotional state information;
wherein the at least one content is associated with the at least one avatar;
electronic device.
9. The method of claim 8,
camera; further comprising,
The instructions, when executed by the processor, cause the electronic device to:
Obtaining the user's gaze information using the camera,
configured to specify a portion of the at least one content based on the gaze information,
electronic device.
9. The method of claim 8,
The instructions, when executed by the processor, cause the electronic device to:
configured to project the identified emotional state onto the at least one avatar;
electronic device.
A method of operating an electronic device, comprising:
executing an application that provides at least one content to the electronic device and the at least one external electronic device;
displaying the at least one content;
specifying a part of the at least one content;
identifying a user's emotional state with respect to the specified content; and
Including; operation of displaying a visual object in association with the specified content based on the identified emotional state
Way.
12. The method of claim 11,
receiving emotional state information identified by the at least one external electronic device with respect to the at least one content; and
Further comprising; updating the visual object based on the received emotional state information;
Way.
12. The method of claim 11,
obtaining user's gaze information; and
Further comprising; specifying a part of the at least one content based on the gaze information;
Way.
13. The method of claim 12,
providing a user interface (UI) related to sharing the emotional state information; and
Further comprising; determining whether to receive or transmit the emotional state information based on the user input to the UI;
Way.
13. The method of claim 12,
Collecting the content identified by the emotional state information (gather) the operation of generating a story (story) content; further comprising,
Way.
13. The method of claim 12,
The application provides a group communication or SNS (social network service) function,
Way.
12. The method of claim 11,
The visual object includes a color, an icon, an emoticon, or an avatar on which the emotional information is projected,
Way.
13. The method of claim 12,
The electronic device is an AR device that provides augmented reality (AR),
outputting at least one avatar related to the application and the at least one content in the augmented reality; and
Further comprising; updating the visual object based on the received emotional state information;
wherein the at least one content is associated with the at least one avatar;
Way.
19. The method of claim 18,
obtaining the user's gaze information; and
Further comprising; specifying a part of the at least one content based on the gaze information;
The electronic device includes a camera set to obtain the user's gaze information,
Way.
19. The method of claim 18.
Projecting the identified emotional state to the at least one avatar; further comprising
Way.

Images