KR20230082374A - An electronic apparatus for adaptive generation of an avatar and a method therefore - Google Patents

Abstract

According to embodiments, an electronic device and an operating method thereof are disclosed. The disclosed electronic device includes a memory for storing one or more instructions; and a processor that executes the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform an avatar mode based on at least one of state information of the user and characteristics of content output from the electronic device. is obtained, the degree of personification of the avatar is determined based on the content characteristics, properties of the avatar are obtained based on the avatar mode and degree of personification, and rendering of the avatar is controlled according to the properties of the obtained avatar.

Description

An electronic apparatus for adaptive generation of an avatar and a method therefore

Various embodiments relate to an electronic device and an operating method thereof, and more specifically, to an electronic device capable of adaptively providing an avatar in consideration of a user's situation and an operating method thereof.

An avatar represents an object to which a user has given his/her appearance in computer technology. For example, the avatar may be expressed in the form of a 2D or 3D model in a computer game or social network service space.

A method for adaptively providing an avatar according to a user's environment is required.

An object of various embodiments is to provide an electronic device capable of generating and providing an avatar based on a user's state or characteristics of content output from the electronic device, and an operating method thereof.

An electronic device according to an embodiment includes a memory for storing one or more instructions; and a processor that executes the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform an avatar mode based on at least one of state information of the user and characteristics of content output from the electronic device. is obtained, the degree of personification of the avatar is determined based on the content characteristics, properties of the avatar are obtained based on the avatar mode and degree of personification, and rendering of the avatar is controlled according to the properties of the obtained avatar.

According to an embodiment, the processor, by executing the one or more instructions, senses at least one of the user's body, posture, eye tracking, and voice characteristics to determine the user's emotion or intention, thereby obtaining the user's state information. can be obtained

According to an embodiment, the processor executes the one or more instructions, so that the avatar mode includes at least one of a companion avatar mode, an assistant avatar mode, and a feature avatar mode, and the state information of the user and the content characteristics Based on at least one of the above, one of the companion avatar mode, the assistant avatar mode, and the feature avatar mode may be acquired as the avatar mode.

According to an embodiment, the processor may determine the avatar mode as the companion avatar mode when the state information of the user represents the user's emotional characteristics by executing the one or more instructions.

According to an embodiment, the processor may determine the avatar mode as the assistant avatar mode when the state information of the user indicates a state in which the user needs help by executing the one or more instructions.

According to an embodiment, the processor may determine the avatar mode as a feature avatar mode based on the characteristics of the content when the state information of the user indicates neutral by executing the one or more instructions.

According to an embodiment, by executing the one or more instructions, the processor may determine a degree of personification of the avatar based on a content characteristic, and determine a higher or lower level of personification of the avatar according to the content characteristic. there is.

According to an embodiment, the attributes of the avatar may include at least one of a character type of the avatar, an appearance of the character, a behavior of the character, properties of an animation performed on the character, or a location where the character is rendered.

According to an embodiment, by executing the one or more instructions, the processor may control rendering of the avatar to be performed on a display of the electronic device or control rendering of the avatar to be performed on an auxiliary device connected through communication. .

According to an embodiment, the auxiliary device includes a head mounted display, and the processor, by executing the one or more instructions, controls the avatar to be rendered in three dimensions (3D) on the head mounted display. can

According to an embodiment, a method of operating an electronic device includes obtaining an avatar mode based on at least one of user status information and content characteristics output from the electronic device, and determining a degree of personification of an avatar based on content characteristics. An operation of acquiring attributes of the avatar based on the avatar mode and degree of personification, and an operation of controlling rendering of the avatar according to the obtained attributes of the avatar.

According to an embodiment, in a computer-readable recording medium on which a program for performing a method of operating an electronic device is recorded, the method of operating the electronic device includes at least one of user state information and content characteristics output from the electronic device. obtaining the avatar mode based on the content characteristics, determining the degree of personification of the avatar based on the characteristics of the content, obtaining properties of the avatar based on the avatar mode and the degree of personification, and according to the obtained properties of the avatar It includes an operation of controlling the rendering of the avatar.

According to various embodiments of the present disclosure, an avatar suitable for at least one of a user's state or content viewed by the user may be appropriately provided to the user.

According to various embodiments of the present disclosure, an avatar suitable for the characteristics of the contents consumed by the user may be appropriately provided to the user.

According to various embodiments of the present disclosure, a more user-friendly avatar can be provided by providing an avatar in consideration of the user's condition including at least one of the user's status and the user's viewing content, which is more positive for the user. experience can be provided.

1 is a reference diagram for explaining a concept of adaptively providing an avatar according to a user's content consumption environment according to various embodiments.
2 is a reference diagram for explaining a system for adaptively providing an avatar according to a user's content consumption environment according to various embodiments.
3 illustrates an example of a block diagram of an electronic device that adaptively provides an avatar according to a user's content consumption environment according to various embodiments.
4 illustrates an example of a detailed block diagram of an electronic device that adaptively provides an avatar according to a user's content consumption environment according to various embodiments.
5 shows an example of an avatar framework according to an embodiment.
6 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.
7 is a reference diagram for explaining a method of acquiring user status information by the user status determination module 510 according to an exemplary embodiment.
8 is a reference diagram for explaining a method of acquiring content characteristic information by the content analysis module 520 according to an exemplary embodiment.
9 is a reference diagram for explaining a method of determining an avatar mode by the avatar mode determination module 530 according to an exemplary embodiment.
10 is a reference diagram for explaining a method of determining the degree of personification of an avatar by the avatar personification degree determining module 550 according to an exemplary embodiment.
11 is a reference diagram for explaining a method of determining an avatar attribute by an avatar attribute determination module 560 according to an exemplary embodiment.
12 is a reference diagram for explaining a method of controlling avatar rendering by the avatar rendering control engine 570 according to an exemplary embodiment.
13 illustrates an example of an operating method of an electronic device operating in companion avatar mode according to an embodiment.
14 illustrates an example of an operating method of an electronic device operating in an assistant avatar mode according to an embodiment.
15 illustrates an example of an operating method of an electronic device operating in an assistant avatar mode according to an embodiment.
16 is a reference diagram for explaining an example in which the electronic device 100 provides an artist avatar according to an embodiment.
17 is a reference diagram for explaining an example in which the electronic device 100 provides a sports player avatar according to an embodiment.
FIG. 18 is a reference diagram for explaining an example in which a user tries on shopping items using an avatar while watching a home shopping channel on the electronic device 100 according to an embodiment.
FIG. 19 is a reference diagram for explaining an example in which the electronic device 100 provides support through an avatar when a problem occurs when a user manipulates or uses the electronic device 100 and needs help to solve a problem, according to an embodiment.
20 is a diagram for explaining a system capable of customizing an avatar according to an embodiment.

The terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected from general terms that are currently widely used as much as possible while considering the functions in the present invention, but these may vary depending on the intention of a person skilled in the art or precedent, the emergence of new technologies, and the like. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, not simply the name of the term.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, with reference to the accompanying drawings, embodiments will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the embodiments of the present disclosure, the term "user" refers to a person who controls a function or operation of a display device using a control device, and may include a viewer, a manager, or an installer.

1 is a reference diagram for explaining a concept of adaptively providing an avatar according to a user's content consumption environment according to various embodiments.

Referring to FIG. 1 , the electronic device 100 may provide content to the user 10 . Also, the electronic device 100 may provide a framework for displaying or controlling avatars to content consuming users.

According to various embodiments, the electronic device 100 may obtain characteristics of the content by analyzing content provided to the user. The content provided to the user by the electronic device 100 may include video content, audio content, or video and audio content. The electronic device 100 may obtain characteristics of content by analyzing using at least one automatic content recognition (ACR) or video/audio scene analysis function.

According to various embodiments, the electronic device 100 may obtain status information of a user from whom content is provided. The electronic device 100 obtains user observation information by detecting the user's body, posture, eye tracking, voice characteristics, etc., and analyzes the obtained user observation information to perform user emotion recognition or user intention recognition to obtain user state information. can be obtained. The electronic device 100 may obtain user observation information through a camera or microphone provided in the electronic device 100 or through an HMD 20 connected to the electronic device 100 and worn by the user.

According to various embodiments, the electronic device 100 may determine an avatar mode based on at least one of user status information and content characteristic information. According to an embodiment, the electronic device 100 may obtain an avatar mode through a pre-stored lookup table (LUT). The lookup table can be dynamically updated based on user interaction. The avatar mode may include at least one of a companion mode, an assistant mode, and a feature mode. The companion mode may indicate an avatar mode capable of acting as a user's friend, colleague, or companion in order to share the user's emotional state. Assistant mode may refer to an avatar mode that can respond to a user's request or provide an answer to a user's request when the user needs some assistance. Assistant mode may also be referred to as Virtual Assistant mode. The feature mode may indicate an avatar mode that may play a role of expanding the content experience based on the characteristics of the content rather than the user's emotional state or the user's request. According to an embodiment, the electronic device 100 may adaptively switch an avatar mode based on user status information or content characteristic information. For example, the electronic device 100 may switch the avatar mode from the companion mode to the assistant mode in consideration of the user's situation. As described above, the electronic device 100 according to various embodiments determines an avatar mode based on at least one of the user's status information and the content's characteristic information, thereby adaptively providing an avatar suitable for the user's situation or user's environment. there is.

According to various embodiments, the electronic device 100 may determine the degree of anthropomorphism of the avatar based on the characteristic information of the content. The degree of anthropomorphism of the avatar may refer to information indicating how realistically the avatar looks and behaves like a human. The higher the degree of anthropomorphism, the more human-like it looks and behaves, and the lower the degree of anthropomorphism, the more distant it seems.

According to various embodiments, the electronic device 100 may obtain attribute information of the avatar based on the avatar mode and the degree of personification of the avatar. The electronic device 100 may obtain avatar attribute information from a pre-stored lookup table based on the avatar mode and the degree of personification of the avatar. The attribute information of the avatar may indicate, for example, the appearance, behavior, personality, and animation characteristics of the avatar character. The avatar may look different and behave differently according to the avatar attributes obtained according to the avatar mode and the degree of personification.

According to various embodiments, the electronic device 100 may control avatar rendering 30 based on avatar attribute information. According to an embodiment, the electronic device 100 may control an avatar to be rendered through the electronic device 100 based on attribute information of the avatar. According to an embodiment, the electronic device 100 may control an avatar to be rendered through an external device connected to the electronic device 100 based on attribute information of the avatar. An external device on which the avatar can be rendered may be, for example, an HMD worn by the user, a user's smartphone, or another display device adjacent to the electronic device 100 . According to an embodiment, the electronic device 100 may control the avatar to be rendered in 2D or 3D when rendering the avatar based on attribute information of the avatar. In particular, when the avatar is rendered through the HMD, the electronic device 100 may control the avatar to be rendered in 3D.

2 is a reference diagram for explaining a system for adaptively providing an avatar according to a user's content consumption environment according to various embodiments.

Referring to FIG. 2 , a system 200 may include an electronic device 100, a head mounted display (HMD) 210, a display device 220, a communication network 230, and an artificial intelligence server 240.

The electronic device 100 may play a role of adaptively providing an avatar according to a user's content consumption environment according to various embodiments disclosed in the present disclosure. The display device 100 may include various types of electronic devices capable of receiving and outputting content, such as, for example, network TV, smart TV, Internet TV, web TV, IPTV, PC, and the like. The electronic device 100 may be referred to as a display device in that it receives and displays content, and may also be referred to as a content receiving device, a sink device, an electronic device, a computing device, and the like.

The head mounted display 210 may transmit and receive data through communication with the electronic device 100 .

According to an embodiment, the head mounted display 210 may obtain user observation information and provide the obtained user observation information to the electronic device 100 . The user observation information may include eye tracking information obtained by tracking the user's eyes or user voice data obtained by receiving the user's voice.

According to an embodiment, the head mounted display 210 may receive avatar rendering information or an avatar rendering command from the electronic device 100 and perform avatar rendering according to the received avatar rendering command or avatar rendering information.

The display device 220 may transmit and receive data through communication with the electronic device 100 .

According to an embodiment, the display device 220 may receive avatar rendering information or an avatar rendering command from the electronic device 100 and perform avatar rendering according to the received avatar rendering command or avatar rendering information. The display device 220 may display a rendering result on a display. The display device 220 can communicate with the electronic device 100 and include various devices having displays. The display device 220 may be the same type of device as the electronic device 100 .

The portable device 230 can communicate with the electronic device 100 and exchange data.

According to an embodiment, the portable device 230 may receive avatar rendering information or an avatar rendering command from the electronic device 100 and perform avatar rendering according to the received avatar rendering command or avatar rendering information. The portable device 230 may display a rendering result on a display. The portable device 230 may include various personal portable devices that can communicate with the electronic device 100 and have a display. For example, the portable device 230 includes a smartphone, a remote controller, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, and a laptop. PC (laptop personal computer), netbook computer, digital camera, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), camcorder, navigation, wearable device, smart watch, home It may include at least one of a network device, a security device, and a medical device.

The artificial intelligence server 240 may provide a response corresponding to the request of the electronic device 100 by using one or more neural network models according to the request of the electronic device 100 . For example, the artificial intelligence server 240 may receive a content analysis request from the electronic device 100, perform content analysis using one or more neural network models, and provide result data to the electronic device 100.

The matching server 250 may receive a fingerprint of the content from the electronic device 100, recognize the content using the received fingerprint, and provide information on the recognized content to the electronic device 100.

3 illustrates an example of a block diagram of an electronic device that adaptively provides an avatar according to a user's content consumption environment according to various embodiments.

Referring to FIG. 3 , an electronic device 100 may include a memory 110 and a processor 120 .

The memory 110 may store programs related to the operation of the electronic device 100 and various data generated during the operation of the electronic device 100 .

The controller 120 may control overall operations of the electronic device 100 by including one or more processors. The controller 120 may adaptively provide an avatar according to the user's content consumption environment by executing one or more instructions stored in the memory 110 .

By executing one or more instructions, the controller 120 according to an embodiment obtains an avatar mode based on at least one of user state information and content characteristics output from the electronic device, and determines the degree of personification of the avatar based on the content characteristics. and the attributes of the avatar are acquired based on the avatar mode and degree of personification, and rendering of the avatar is controlled according to the obtained attributes of the avatar.

The control unit 120 according to an embodiment obtains the user's state information by sensing at least one of the user's body, posture, eye tracking, and voice characteristics to determine the user's emotion or intention by executing the one or more instructions. can do.

The controller 120 according to an embodiment executes the one or more instructions, so that the avatar mode includes at least one of a companion avatar mode, an assistant avatar mode, and a feature avatar mode, and among the state information of the user and the content characteristics Based on at least one, one of the companion avatar mode, the assistant avatar mode, and the feature avatar mode may be obtained as the avatar mode.

The controller 120 according to an embodiment may determine the avatar mode as the companion avatar mode when the user state information indicates the user's emotional characteristics by executing the one or more instructions.

The controller 120 according to an embodiment may determine the avatar mode as the assistant avatar mode when the user state information indicates a state in which the user needs help by executing the one or more instructions.

According to an embodiment, the controller 120 may determine the avatar mode as a feature avatar mode based on the characteristics of the content when the status information of the user indicates neutral by executing the one or more instructions.

By executing the one or more instructions, the controller 120 according to an embodiment may determine the degree of personification of the avatar based on the content characteristics, and may determine the strength of the personification of the avatar to be higher or lower according to the characteristics of the content. .

According to an embodiment, the attributes of the avatar may include at least one of a character type of the avatar, an appearance of the character, a behavior of the character, properties of an animation performed on the character, or a location where the character is rendered.

By executing the one or more instructions, the controller 120 according to an embodiment may control rendering of the avatar to be performed on the display of the electronic device or control rendering of the avatar to be performed on an auxiliary device connected through communication.

4 illustrates an example of a detailed block diagram of an electronic device that adaptively provides an avatar according to a user's content consumption environment according to various embodiments.

Referring to FIG. 4 , the electronic device 100 includes, in addition to a memory 110 and a control unit 120, a display 130, a video processing unit 135, an audio output unit 140, an audio processing unit 145, a communication unit 150, a sensing unit 160, a user input unit 170, an input/output unit 180, An avatar framework 500 may be further included.

The display 130 may display the video signal received from the video processing unit 135 on the screen. The display 130 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, and a 4D display. , at least one of electrophoretic displays.

The video processor 135 may receive video signals from various sources under the control of the controller 120, process the received video signals, and provide the received video signals to the display 130.

The audio output unit 140 may output a received analog audio signal through a speaker.

The audio processor 145 may convert an audio signal received from the input/output unit 180 or the communication unit 150 into an analog audio signal and output the analog audio signal to the audio output unit 140 under the control of the controller 120 .

The communication unit 150 may include one or more modules enabling wireless communication between the electronic device 100 and a wireless communication system or between the electronic device 100 and a network where other electronic devices are located. For example, the communication unit 150 may include a broadcast reception module 151, a mobile communication module 152, a wireless Internet module 153, and a short-distance communication module 154.

The mobile communication module 152 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include a voice call signal, a video call signal, or various types of data according to text/multimedia message transmission/reception.

The wireless Internet module 153 refers to a module for wireless Internet access, and may be built into or external to a device. Wireless Internet technologies include wireless LAN (WLAN) (WiFi), wireless broadband (Wibro), world interoperability for microwave access (Wimax), high speed downlink packet access (HSDPA), and the like. Through the wireless Internet module 153, the device can establish a Wi-Fi peer to peer (P2P) connection with other devices.

According to an example, the wireless Internet module 153 may communicate with the artificial intelligence server 240 or the matching server 250 according to the control unit 120 .

The short-distance communication module 154 refers to a module for short-distance communication. Bluetooth, Bluetooth Low Energy (BLE), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like may be used as short-distance communication technologies.

According to an example, the short-distance communication module 154 may communicate with one or more of the HMD 210, the display device 220, and the portable device 230 under the control of the controller 120 to transmit and receive data.

The sensing unit 160 senses a user's voice, a user's video, or a user's interaction, and may include a microphone 161, a camera unit 162, and a light receiving unit 163.

The microphone 161 may receive a user's utterance and convert the received voice into an electrical signal to output to the control unit 120 .

The camera unit 162 includes a sensor and a lens, and may capture an image formed on the screen.

The light receiving unit 163 may receive an optical signal. The light receiving unit 163 may receive an optical signal corresponding to a user input (eg, touch, pressing, touch gesture, voice, or motion) from a control device such as a remote control or a mobile phone.

The user input unit 170 means a means through which the user inputs data for controlling the electronic device 100 . For example, the user input unit 170 may include, but is not limited to, a key pad, a dome switch, a touch pad, a jog wheel, and a jog switch.

The input/output unit 180 transmits video (eg, motion picture, etc.), audio (eg, voice, music, etc.) and additional information (eg, EPG, etc.) from the outside of the electronic device 100 under the control of the controller 120. etc. can be received. The input/output unit 180 includes one of an HDMI port (High-Definition Multimedia Interface port, 181, a component jack, 182), a PC port (PC port, 183), and a USB port (USB port, 184), or In addition to the HDMI port, the input/output unit 180 may further include DisplayPort (DP), Thunderbolt, and Mobile High-Definition Link (MHL).

The memory 120 may store programs related to the operation of the display device 100 and various data generated during operation of the display device 100 .

The avatar framework 500 may include appropriate logic, circuitry, interfaces, and/or code that may operate to provide an avatar suitable for a user's condition or environment according to embodiments disclosed in this disclosure. The avatar framework 500 is executed as a background process during the operation of the electronic device 100, and may be triggered and executed by a user's speech or the like, or may be executed according to a user's request. The avatar framework 500 will be described in detail with reference to FIG. 5 .

5 shows an example of an avatar framework according to an embodiment.

Referring to FIG. 5 , according to an embodiment, the avatar framework 500 includes a user state determination module 510, a content analysis module 520, an avatar mode determination module 530, an avatar mode lookup table 540, an avatar personification strength determination module 550, and an avatar attribute determination module. 560, and an avatar rendering control engine 570.

The user state determination module 510 may include appropriate logic, circuits, interfaces, and/or codes operable to obtain state information of a user consuming content provided by the electronic device 100 . The user status determination module 510 may provide the obtained user status information to the avatar mode determination module 530 . The user state determination module 510 acquires user observation information by detecting the user's body, posture, eye tracking, voice characteristics, etc., and analyzes the obtained user observation information to perform user emotion recognition or user intention recognition to determine the user's status information can be obtained. The user state determination module 510 may obtain user observation information through a camera or a microphone, or may acquire user observation information through an HMD 210 connected to the electronic device 100 and worn by the user.

The content analysis module 520 may include suitable logic, circuitry, interfaces, and/or code operable to analyze content provided to a user and obtain characteristics of the content. The content analysis module 520 may provide the obtained content characteristic information to the avatar mode determination module 530 . The content provided to the user by the electronic device 100 may include video content, audio content, or video and audio content. The content analysis module 520 may acquire characteristics of content by analyzing using at least one automatic content recognition (ACR) or video/audio scene analysis function.

The avatar mode determination module 530 may determine an avatar mode based on at least one of the user state information received from the user state determination module 510 and content characteristic information received from the content analysis module 520, suitable logic, circuitry, and It may contain interfaces and/or code. The avatar mode determination module 530 may provide the determined avatar mode to the avatar attribute determination module 560 . According to an embodiment, the avatar mode determination module 530 may obtain an avatar mode through a pre-stored lookup table (LUT). The lookup table can be dynamically updated based on user interaction. The avatar mode may include at least one of a companion mode, an assistant mode, and a feature mode. According to an embodiment, the avatar mode determination module 530 may adaptively switch an avatar mode based on user state information or content characteristic information. For example, the avatar mode determination module 530 may switch the avatar mode from the companion mode to the assistant mode in consideration of the user's situation. As described above, the avatar mode determination module 530 according to various embodiments determines an avatar mode based on at least one of the user's state information and content characteristic information, thereby adaptively providing an avatar suitable for the user's situation or user's environment. can do.

The avatar anthropomorphism strength determination module 550 may include suitable logic, circuitry, interfaces and/or codes operable to determine the degree of anthropomorphism of the avatar based on the content characteristic information received from the content analysis module 520. . The degree of anthropomorphism of the avatar may refer to information indicating how realistically the avatar looks and behaves like a human. The higher the degree of anthropomorphism, the more human-like it looks and behaves, and the lower the degree of anthropomorphism, the more distant it seems. The avatar personification strength determination module 550 may provide the determined avatar personification strength to the avatar attribute determination module 560 .

The avatar attribute determination module 560 may include suitable logic, circuitry, interfaces, and logic operable to obtain avatar attribute information based on the avatar mode received from the avatar mode determination module 530 and the degree of avatar anthropomorphism received from the avatar anthropomorphism strength determination module 550. /or may contain code. The avatar attribute determination module 560 may provide the obtained avatar attribute information to the avatar rendering control engine 570 . The avatar attribute determination module 560 may obtain attribute information of the avatar based on the avatar mode and the degree of personification of the avatar from a pre-stored lookup table. The attribute information of the avatar may indicate, for example, the appearance, behavior, personality, and animation characteristics of the avatar character. The avatar may look different and behave differently according to the avatar attributes obtained according to the avatar mode and the degree of personification.

The avatar rendering control engine 570 may include suitable logic, circuitry, interfaces and/or codes operable to control rendering of an avatar based on avatar property information received from the avatar property determination module 560 . According to an embodiment, the avatar rendering control engine 570 may control an avatar to be rendered through the electronic device 100 based on attribute information of the avatar. According to an embodiment, the avatar rendering control engine 570 may control an avatar to be rendered through an external device connected to the electronic device 100 based on attribute information of the avatar. An external device on which the avatar can be rendered may be, for example, the HMD 210 worn by the user, another display device 220 adjacent to the electronic device 100, or the user's portable device 230.

6 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

Referring to FIG. 6 , in operation 610, the electronic device 100 may acquire an avatar mode based on at least one of user state information and content characteristic information output to the electronic device 100.

In detail, user status information may be obtained by the user status determination module 510 of the electronic device 100 . A method of acquiring user status information will be described with reference to FIG. 7 .

7 is a reference diagram for explaining a method of acquiring user status information by the user status determination module 510 according to an exemplary embodiment.

A user state may represent a user's emotion or intention. In order to understand the user 20 , the user state determination module 510 may obtain user observation information 710 including at least one of facial expression, body, posture, eye tracking, and voice characteristics of the user.

According to an embodiment, the user state determination module 510 may capture the user's expression, body, or posture using the camera of the electronic device 100 . According to an embodiment, the user state determination module 510 may receive user voice data using the microphone of the electronic device 100 .

According to an embodiment, the user state determination module 510 may receive user observation information 710 from an external device connected to the electronic device 100, for example, the HMD 210. For example, the user state determination module 510 may receive eye tracking information, voice data, and hand tracking information acquired through an eye tracking sensor, a microphone, or a hand tracking sensor of the HMD 210 .

According to an embodiment, the user state determination module 510 may determine the user's emotion or intention 720 based on the user observation information 710 collected in this way. The user state determination module 510 may determine the user's intention or emotion based on the user observation information using one or more artificial intelligence models and may determine the user state information 730 . The user state information may include one or more state information such as a first state, a second state, and a third state. The user state determination module 510 may determine one of a plurality of user states based on user observation information. For example, based on eye-tracking data, an artificial intelligence model can classify a user's emotion as feeling lonely. In another embodiment, through multi-modal data analysis and classification, the artificial intelligence model can classify the user's intent as requiring support. For example, AI models can classify user status into three categories: feeling lonely, needing help, or neutral. As will be described below, if the user status is classified as feeling lonely, the mode of the avatar may be determined as a companion. When the user status is determined to require assistance, the mode of the avatar may be determined to be an assistant mode. If the user state is determined to be feeling neutral, the avatar maintains the previous mode or the avatar mode may be determined based on analysis of TV content. The relationship between the user state information and the optimal avatar mode may be stored in a predefined LUT.

Returning to FIG. 6 , the electronic device 100 may acquire content characteristic information output from the electronic device 100 . In detail, the content characteristic information may be acquired by the content analysis module 520 of the electronic device 100 . A method of obtaining content property information will be described with reference to FIG. 8 .

8 is a reference diagram for explaining a method of acquiring content characteristic information by the content analysis module 520 according to an exemplary embodiment.

According to an embodiment, the content analysis module 520 may extract a content fingerprint from the audio and/or video content 810 output from the electronic device 100 and deliver the extracted fingerprint to the matching server 250 . The matching server 250 uses the content fingerprint received from the electronic device 100 to recognize content using Automatic Content Recognition (ACR) technology, etc., and provides information on the recognized content or metadata about the recognized content. It can be transmitted to the electronic device 100 . The content analysis module 520 may obtain the content characteristic 830 by using the received content information or content metadata 820 .

According to an embodiment, the content analysis module 520 may obtain characteristics of audio and/or video content using a pre-trained AI model. The pre-trained AI model may be stored in the electronic device 100 or the content analysis module 520 transmits part of the audio and/or video content to the artificial intelligence server 240 through the communication network 260, thereby providing characteristic information of the content from the artificial intelligence server 240. 830 can be obtained.

According to an embodiment, the content analysis module 520 may obtain metadata of audio and/or video content output from the electronic device 100 and obtain content property information 830 from the metadata.

The content characteristic information 830 may include, for example, the type or genre of content, the mood of content, and age restriction information of content. For example, the type or genre of content may include games, sports, dramas, sound wall apps, and the like. For example, the mood of the content may indicate whether the mood of the content is sad, happy, or violent. Age restriction information of the content may include information about the viewing age of the content.

Returning to FIG. 6 , the electronic device 100 may acquire an avatar mode based on at least one of user status information and content characteristic information. In detail, the avatar mode may be acquired by the avatar mode determination module 530 of the electronic device 100 . A method of determining an avatar mode will be described with reference to FIG. 9 .

9 is a reference diagram for explaining a method of determining an avatar mode by the avatar mode determination module 530 according to an exemplary embodiment.

According to an embodiment, the avatar mode determination module 530 may determine an avatar mode based on at least one of user state information and content characteristic information. In this case, the avatar mode determination module 530 may determine the avatar mode by referring to the avatar mode lookup table 540 .

The avatar mode may be determined in various ways, but in the example illustrated in FIG. 9 , the avatar mode may include a companion avatar mode, an assistant avatar mode, and a feature avatar mode. The companion avatar mode represents an avatar mode serving as a companion next to the user. In the companion avatar mode, companion avatars, for example, avatars such as friends or co-workers are created and can play a role in exchanging conversations with the user. The assistant avatar mode represents an avatar mode that serves to execute a user's request or command according to a user's request or command. In the assistant avatar mode, an assistant avatar, for example, a personal assistant, may be created and perform a user's request, instruction, or command. The feature avatar mode is created based on the characteristics of the content independently of the user's state, and can play a role in enriching the experience of the content. For example, in the feature avatar mode, when the content property is music content, the feature avatar plays the role of a character playing music content, or when the content property is sports content, the feature avatar plays the role of a character cheering for sports. can

The avatar mode determination module 530 may determine an avatar mode by referring to the avatar mode lookup table 540 according to at least one of user state information and content characteristic information. This avatar lookup table 540 may be designed in advance by a developer and may be dynamically updated according to user interaction or user preference.

According to an example, the avatar mode determination module 530 may determine the avatar mode as the companion mode when the user status information indicates a certain emotional characteristic. When the user state information indicates that a certain emotional factor of the user is high, such as joy, sadness, or loneliness, it may be determined that a companion is needed to share the user's emotions. Accordingly, the avatar mode determining module 530 may determine the companion avatar mode when the user status information indicates emotional characteristics.

According to an example, the avatar mode determination module 530 may determine the avatar mode as an assistant mode when the user status information indicates that help or support is needed. When the user status information indicates a state in which the user needs some help rather than an emotional aspect, it may be determined that an assistant to solve the problem is needed. Accordingly, the avatar mode determining module 530 may determine the assistant avatar mode when the user status information indicates that assistance is required.

According to an example, the avatar mode determination module 530 may determine the avatar mode by referring to the content characteristic information when the user state information indicates a neutral feeling. A companion or assistant may not be needed if the user's status information is not a specific emotional aspect or a condition requiring specific help. Accordingly, in this case, the avatar mode determination module 530 may determine the feature avatar mode based on content characteristic information rather than user state information. For example, when the user status information indicates neutral feeling and the content property information indicates sports as the content genre, the avatar mode determining module 530 may determine the avatar mode as the companion avatar mode. For example, when the user status information indicates a neutral feeling and the content property information indicates a sound wall application as the content genre, the avatar mode determining module 530 may determine the avatar mode as a feature avatar mode.

Returning to FIG. 6 again, in step 620, the electronic device 100 may determine the degree of personification of the avatar based on the content characteristics. In detail, the avatar personification degree may be obtained by the avatar personification degree determining module 550 of the electronic device 100 . A method of determining the degree of personification of the avatar will be described with reference to FIG. 10 .

10 is a reference diagram for explaining a method of determining the degree of personification of an avatar by the avatar personification degree determining module 550 according to an exemplary embodiment.

Some experimental studies show that the type of content can affect an avatar's perception. For example, when people receive support or direction from a more realistic and human-looking avatar while playing a game, they can usually feel negative emotions because they feel that their control or authority over the game is being undermined by the avatar. However, in other scenarios or content, people may feel positive emotions when seeking information or help as AI assistants that look more realistic and human appear more trustworthy. Therefore, it may be important to adapt the avatar according to the characteristics of the content.

Referring to FIG. 10 , the avatar personification degree determination module 530 may adjust the avatar personification degree according to content characteristics. For example, the avatar's anthropomorphism may be classified as low, medium, or high. The lower the degree of personification of the avatar, the less realistic or less human the avatar is, and the higher the degree of personification of the avatar, the more realistic or more human. For example, an avatar having a low level of avatar anthropomorphism may represent an avatar that personifies an object or a plant that is far from a person. For example, an avatar having a medium level of anthropomorphism may represent an avatar that anthropomorphizes an animal character similar to a human or an animation character. For example, an avatar having a high level of avatar personification may represent an avatar close to a realistic person.

For example, in the assistant avatar mode, the avatar may have a high degree of anthropomorphism when the nature of the content provides explanations of TV functions or solves problems. Even in the assistant mode, when the characteristics of the content are game content, the avatar may have a low degree of personification when assisting the user. For example, when the characteristics of the content indicate a genre such as cartoon, the degree of personification of the avatar may be medium. In this way, the relationship between the characteristics of content and the degree of personification may be determined through research, user research, and pilot testing, and may be pre-stored in a lookup table. Also, these lookup tables can be updated according to the user's personal preferences.

Returning to FIG. 6 again, in the child 630, the electronic device 100 may acquire attributes of the avatar based on the avatar mode and the degree of personification of the avatar. Specifically, the attributes of the avatar may be obtained by the avatar attribute determination module 560 of the electronic device 100 . A method of acquiring avatar attributes will be described with reference to FIG. 11 .

11 is a reference diagram for explaining a method of determining an avatar attribute by an avatar attribute determination module 560 according to an exemplary embodiment.

Referring to FIG. 11 , the avatar attribute determination module 560 may acquire avatar attributes based on an avatar mode 900 and an avatar personification degree 1000.

The avatar attribute determination module 560 may acquire avatar attributes based on the avatar mode 900 and the degree of personification of the avatar 1000. The avatar properties may include, for example, the avatar's character, the avatar's appearance, the avatar's behavior, and the avatar's animation properties. Depending on the attributes of the avatar, the avatar may look different and behave differently.

For example, when the avatar mode 900 represents a companion avatar mode and the content characteristic indicates that the degree of personification of the avatar is high according to a genre such as a movie, the avatar attribute determination module 560 determines the avatar attribute 1100 as a more realistic and human-looking avatar. You can decide by character. In addition, to complete a human-looking avatar character, the avatar attribute determination module 560 may determine the avatar's clothing, style, and appearance to look more human, and may also determine animation attributes to look more human. For example, if the avatar mode indicates the companion avatar mode and the content characteristic indicates that the degree of avatar anthropomorphism is low according to the game genre, the avatar attribute determination module 560 selects the avatar character as a less human character and makes it behave less human. and may decide to have animation characteristics less human-like. For example, in this case, the avatar character has the shape of a car and may have animation characteristics similar to the shape of the car.

According to an embodiment, when an auxiliary device can display an avatar in mixed reality (MR), the position of the avatar may be a property of the avatar. For example, the auxiliary device may be an HMD or a mobile device. For example, when the avatar is in the companion mode, the avatar's location range may appear near the user, for example, the avatar may be placed on a sofa next to the user. In another embodiment, when the avatar is in assistant mode, the avatar's location may be limited to a TV away from the user, and may be viewed only when called by the user. In another embodiment, the avatar in the companion mode may frequently change its location and perform animation, but the avatar in the assistant mode may not automatically change its location unless requested by the user.

Returning to FIG. 6 again, in step 640, the electronic device 100 may control rendering of the avatar according to the acquired attributes of the avatar. In detail, avatar rendering control may be acquired by the avatar rendering control engine 570 of the electronic device 100 . A method of controlling rendering of an avatar will be described with reference to FIG. 12 .

12 is a reference diagram for explaining a method of controlling avatar rendering by the avatar rendering control engine 570 according to an exemplary embodiment.

Referring to FIG. 12 , the avatar rendering control engine 570 may control avatar rendering based on avatar attributes. In detail, the avatar rendering control engine 570 may control rendering of the avatar on the display of the electronic device 100, or avatar rendering information or avatar rendering commands may be performed in an external device connected to the electronic device 100 other than the electronic device 100. can be transmitted to an external device.

According to an embodiment, the avatar rendering control engine 570 may render the avatar through a multi-view window of a display provided in the electronic device 100 or a PIP (Picture in Picture). The size and position of the avatar can be optimally adjusted so that it can be presented in a way convenient for the user. The avatar rendering control engine 570 may adjust the avatar's sound or hearing characteristics according to the avatar's properties, environment, TV volume setting, and user's preference. In addition, the avatar rendering control engine 570 may continuously adjust the avatar rendering according to the user's location, the state of the second device, environment understanding, device complexity, and the like.

According to an embodiment, the avatar rendering control engine 570 may provide avatar rendering information or an avatar rendering command to render an avatar to another display device having a camera. A display device with a camera can capture a real-world environment and an avatar can be superimposed on this captured real-world view to provide a mixed reality (MR) feeling without an HMD or mobile device.

According to an embodiment, when an external device connected to the electronic device 100 is an HMD, the HMD may display an avatar overlaid on mixed reality (MR), that is, the user's real world. The HMD can utilize eye tracking, spatial mapping and mixed reality capture to properly render the avatar in the environment. For example, the HMD may divide the position of the electronic device 100 in the environment and render the position of the avatar in a manner that does not obstruct the user's view of the electronic device 100 .

13 illustrates an example of an operating method of an electronic device operating in companion avatar mode according to an embodiment. In the example shown in FIG. 13 , a TV will be described as an example of an electronic device.

Referring to FIG. 13 , in operation 1301, the user can watch TV. A TV is an example of an electronic device 100 capable of providing an avatar suitable for a user's situation by storing an avatar framework according to various embodiments. The TV can output content.

In operation 1302, the user wears the HMD, and the HMD and TV can be paired and connected. The HMD may function as an auxiliary device according to various embodiments, and when the user wears the HMD, the TV may be paired with the HMD.

In operation 1303, the user may utter "I want to watch TV with a friend". Upon receiving such a user's speech, the TV recognizes the voice corresponding to the user's speech through an automatic speech recognition (ASR) engine and interprets the user's speech through a natural language analysis module (NLU) to acquire the user's state information. can That is, the TV can determine the user's state (intention) as requiring a companion through natural language interpretation of the user's utterance, and determine the companion avatar mode. And, the TV can determine the avatar attribute by determining the degree of personification of the avatar based on the characteristics of the content output from the TV.

In operation 1304, the TV may transmit avatar rendering information or an avatar rendering command to the HMD to show various humanoid avatars (highly anthropomorphic) determined according to avatar attributes to the user. The HMD receiving the avatar rendering information may output various humanoid avatars, and the user may select a desired avatar through hand gesture interaction with the HMD. Some attributes of the selected avatar depend on the avatar mode, in which case it may depend on the companion mode and degree of personification.

In operation 1305, the HMD may render and control the selected avatar according to the user's avatar selection. For example, the avatar may appear on a sofa next to the user. The avatar can always be seen when it is in the user's view (FOV), or it can only be seen when the user wearing the HMD heads toward the sofa.

In operation 1306, content including an interesting scene may be output on the TV. The TV analyzes the contents of the output content (criterion for understanding the scene), determines that the scene is funny, and obtains attribute information of the avatar from a pre-stored lookup table based on the analyzed content. For example, TV can acquire laughter as an animation attribute of an avatar. The TV may transmit the animation properties of the newly acquired avatar to the HMD.

In operation 1307, the HMD may render the animation of the avatar set as a smile according to the avatar attribute received from the TV.

At action 1308, the user says "Funny, do you know who that actress is?" can be ignited. The TV can understand the user's question by receiving a voice corresponding to the user's speech and interpreting the spoken content. In addition, in order to prepare an answer to the user's question, the TV analyzes the output TV content using Automatic Content Recognition (ACR), and based on the analyzed content, the text corresponding to the answer to the user's question is converted into an avatar. can be determined as a property of The TV may transmit text corresponding to an answer to the user's question as an attribute of the avatar to the HMD.

In operation 1309, the HMD converts the response text as an attribute of the avatar received from the TV into voice and renders the avatar to be uttered. For example, the avatar may reply "Uh, she's a French model and actress."

In operation 1310, when the user makes a high-five gesture, the TV captures the user's gesture with a camera, recognizes the user's gesture, and determines the user's status as a high-five interaction. The TV may determine the high-five action as an animation property of the avatar based on the user's state, and transmit the determined property of the avatar to the HMD. The HMD renders the avatar so that the avatar's hand can touch the user's hand by rendering and controlling the avatar based on the high-five motion as an attribute of the avatar received from the TV, so that the user can experience the high-five with the avatar.

In the example shown in FIG. 13 , it has been described that the avatar framework is installed in the TV and the TV plays the main function for avatar rendering control. However, some or all of the functions of the avatar framework may also be implemented in the HMD. Accordingly, the implementation of the avatar as described in FIG. 13 may be realized by mutual cooperation between the TV and the HMD.

14 illustrates an example of an operating method of an electronic device operating in an assistant avatar mode according to an embodiment.

Referring to FIG. 14 , in operation 1401, the user may say “Turn on the light”. Upon receiving the voice data corresponding to the user's speech, the electronic device 100 can understand the user's speech through analysis such as voice recognition and natural language understanding. The electronic device 100 may determine from the contents of the speech that the user is in a state in which help is needed, and accordingly, the electronic device 100 may determine the avatar mode as the assistant avatar mode. In addition, information for performing an answer and an action corresponding to the content of the user's utterance may be determined as an attribute of the avatar. For example, as an attribute of the avatar, the electronic device 100 prepares an answer of "yes" as an answer corresponding to the content of the user's speech, and also controls lighting as an operation corresponding to the content of the user's speech. You can send a command to turn on the lights.

In operation 1402, when the electronic device 100 transmits avatar rendering information and an answer of “yes” to the HMD connected through communication, the HMD renders the avatar according to the received avatar rendering information and the answer of “yes”, and also in operation 1403, as an avatar. You can control it to answer "yes".

In operation 1404, the HMD controls rendering so that the avatar walks toward the light, and in operation 1405, the HMD controls rendering so that the avatar makes a gesture to turn on the light.

In operation 1406, the electronic device 100 transmits a control signal to turn on the light to the lighting device connected through communication, and the lighting device can turn on the light according to the control signal.

In operation 1407, the HMD may control rendering so that the avatar that has completed the lighting-up gesture makes a returning gesture.

In operation 1408, the HMD controls rendering so that the returned avatar utters "I'm done", and in operation 1409, the avatar may disappear.

In operation 1410, the electronic device 100 may update and display the turned-on state of the communication-connected lighting device on the SmartThings dashboard.

15 illustrates an example of an operating method of an electronic device operating in an assistant avatar mode according to an embodiment. In the example shown in FIG. 15 , a TV will be described as an example of an electronic device.

Referring to FIG. 15 , in operation 1501, the user may utter "notify me when this program starts". Upon receiving the voice data corresponding to the user's speech, the electronic device 100 can understand the user's speech through analysis such as voice recognition and natural language understanding. The electronic device 100 may determine from the contents of the speech that the user is in a state in which help is needed, and accordingly, the electronic device 100 may determine the avatar mode as the assistant avatar mode. In addition, information for performing an answer and an action corresponding to the content of the user's utterance may be determined as an attribute of the avatar. For example, as an attribute of the avatar, the electronic device 100 prepares an answer "Yes, I'll set a reminder" as an answer corresponding to the user's speech content, and also reserves a program requested by the user as an action corresponding to the user's speech content. It can be registered as a setup program.

In operation 1502, the electronic device 100 displays an avatar on a part of the display of the electronic device 100, and in operation 1503, the electronic device 100 may control rendering so that the avatar responds, "Yes, I'll set a reminder."

In operation 1504, the electronic device 100 may control rendering so that the avatar disappears from the display.

In operation 1505, the electronic device 100 may control rendering so that the avatar is displayed on the display when the program start time requested by the user arrives.

In operation 1506, the electronic device 100 may control rendering so that the avatar utters "It's time to watch this program."

In operation 1507, the electronic device 100 may render and control an animation in which the avatar presses the remote control.

In operation 1508, the electronic device 100 converts the program into a channel corresponding to the program requested by the user and controls the corresponding program to be output.

In operation 1509, the electronic device 100 may control rendering so that the avatar disappears from the display.

In another embodiment, the user may request "show me interesting content". Based on the state of the user needing help, the electronic device 100 may determine the avatar mode as the assistant avatar mode. The electronic device 100 may generate a list of content that may be of interest to the user based on a TV viewing history or user preference, and provide the created content list along with an avatar to the user through a display or HMD of the electronic device 100 . According to a user input for selecting one of the provided contents, the electronic device 100 may reproduce the selected contents.

In another embodiment, when a hearing-impaired person requests help with a TV function or an electronic manual stored in the TV, the electronic device 100 may determine the avatar mode as an assistant avatar mode. The electronic device 100 may explain TV functions or electronic manuals using pre-stored avatars through sign language animations. Also, the audio content of the TV may be converted into text, and the electronic device 100 may control rendering so that the assistant avatar performs a sign language animation that can be presented to the hearing-impaired or hearing-impaired user.

According to an embodiment, the electronic device 100 may provide an artist avatar in synchronization with music content displayed on a display.

16 is a reference diagram for explaining an example in which the electronic device 100 provides an artist avatar according to an embodiment. In FIG. 16 , a TV is described as an example of the electronic device 100 .

Referring to FIG. 16 , the electronic device 100 may execute music content, eg, a music wall application, and output a background screen displaying music and a dynamic pattern. While the electronic device 100 is outputting music content, the electronic device 100 may determine a feature avatar mode corresponding to the music content being output.

The electronic device 100 acquires the genre of the music content, the performer of the music content, the atmosphere of the music content, etc. by analyzing the metadata of the music content or the audio of the music content output to the display, and based on the obtained characteristics, the electronic device 100 acquires the artist. You can decide your avatar character. For example, when the music content is a song such as a popular song or a pop song, the electronic device 100 may determine the singer of the song as the avatar character. For example, when the music content is a piece of music, the electronic device 100 may determine an avatar character such as a performer or an instrument played by the performer.

The electronic device 100 displays the created avatar character on a partial area of the display of the electronic device 100 or outputs it to the HMD connected to the electronic device 100, so that while viewing music content through the electronic device 100, the artist related to the corresponding music content through the HMD. Avatar character 1600 can be controlled to be rendered.

According to an embodiment, the electronic device 100 may provide sports player avatars in synchronization with sports content displayed on the display.

17 is a reference diagram for explaining an example in which the electronic device 100 provides a sports player avatar according to an embodiment. In FIG. 17 , a TV is described as an example of the electronic device 100 .

Referring to FIG. 17 , the electronic device 100 may output sports content. While the electronic device 100 outputs sports content, the electronic device 100 may determine a feature avatar mode in response to the output sports content.

The electronic device 100 may determine the type of sport by analyzing the sports content output on the display using ACR or scene recognition technology. That is, the electronic device 100 determines whether the content displayed on the display is soccer, basketball, or baseball, recognizes a player appearing in the sports content, and creates an avatar character 1700 corresponding to one of the recognized players. can create

The electronic device 100 outputs the created avatar character to the HMD connected to the electronic device 100, so that the user can control rendering of player avatar characters appearing in the sports content through the HMD while watching sports content through the electronic device 100. there is.

According to an embodiment, the electronic device 100 may provide an avatar for trying on shopping items when displaying home shopping channel content. The electronic device 100 may determine the feature avatar mode according to the characteristics of the home shopping channel content, or may determine the assistant avatar mode according to the user's intention to try on an item shown in the home shopping channel content.

FIG. 18 is a reference diagram for explaining an example in which a user tries on shopping items using an avatar while watching a home shopping channel on the electronic device 100 according to an embodiment. 18 illustrates a TV as an example of the electronic device 100 .

Referring to FIG. 18 , in operation 1810, the electronic device 100 may output content corresponding to a home shopping channel. A user may want to try on shopping items while watching home shopping channel content.

At operation 1820, the user may wear the AR glasses.

In operation 1830, the electronic device 100 receives voice data corresponding to the user's speech as the user indicates the user's intention by saying, for example, "I want to try on that shopping item." can understand the contents of the user's speech through analysis such as voice recognition and natural language understanding. The electronic device 100 may determine from the contents of the speech that the user is in a state in which help is needed, and accordingly, the electronic device 100 may determine the avatar mode as the assistant avatar mode. Alternatively, the electronic device 100 may determine a feature avatar mode in response to home shopping channel content. Accordingly, the electronic device 100 may capture the user's image as an avatar attribute and reflect the captured image to create an avatar character representing the user's own appearance. The electronic device 100 may output images of the created avatar character and shopping items output from the electronic device 100 to AR glasses connected through a communication connection.

In operation 1840, the user may select a desired shopping item from among shopping items output through the AR glasses.

In operation 1850, the AR glasses may render an animation for virtually fitting the shopping item selected by the user to the user's own avatar according to the user's selection.

In FIG. 18, it has been described that the user's own avatar character is displayed on the AR glasses connected by communication, but the avatar character is displayed on the display of the electronic device 100, and animation through virtual fitting may also be rendered on the display of the electronic device 100.

According to an embodiment, the electronic device 100 may provide an avatar when a problem occurs when a user manipulates or uses the electronic device 100 and needs help to solve a problem. At this time, the avatar may serve as a virtual assistant. When the electronic device 100 is a TV, the TV analyzes various context data related to the TV, such as audio volume, external device connection, audio settings, picture settings, etc., and retrieves an appropriate text response from an electronic manual (e-manual) stored in advance in the TV. can do. Text responses can be converted into speech using the Text-to-Speech (TTS) model and presented to the user through avatar animations and dialogues. That is, the TV can provide help and support for TV problems to the user through an interactive avatar in a virtual assistant avatar mode.

FIG. 19 is a reference diagram for explaining an example in which the electronic device 100 provides support through an avatar when a problem occurs when a user manipulates or uses the electronic device 100 and needs help to solve a problem, according to an embodiment. In FIG. 19 , a TV is described as an example of the electronic device 100 .

While viewing content through the electronic device 100, the user may indicate that there is a problem because there is a problem with the sound of the content. For example, the user may say "TV volume level is 100, but no sound can be heard", and the electronic device 100 may receive such speech in operation 1910. Upon receiving the voice data corresponding to the user's speech, the electronic device 100 can understand the user's speech through analysis such as voice recognition and natural language understanding. The electronic device 100 can determine from the contents of the speech that the user is in need of help, and accordingly, the electronic device 100 can determine the avatar mode as the assistant avatar mode. Information for performing answers and actions corresponding to the content of the user's speech may be determined as attributes of the avatar.

The electronic device 100 can find a solution corresponding to the user's problem by searching an electronic manual that stores TV problems and corresponding solutions by interpreting the user's speech. Also, the electronic device 100 may determine the content of an answer corresponding to the found solution as an attribute of the avatar. In the next operation 1920, the electronic device 100 may display the avatar 1930 on the display. In operation 1940, the electronic device 100 may render the avatar displayed on the display to utter "Please turn up the volume with the set-top box remote control", which is an answer corresponding to the previously found solution.

20 is a diagram for explaining a system capable of customizing an avatar according to an embodiment.

Referring to FIG. 20 , according to avatar sponsorship 2001, a plurality of brands 2002 and a content producer 2003 may provide various avatar contents to 2004 through NFT market play. Various avatar contents may include an avatar character, a dress to be worn by the avatar character, and accessories.

The user can perform his/her own avatar customization 2005 by accessing the NFT marketplace 2004 and purchasing avatar contents desired by the user.

An operating method of an electronic device according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also within the scope of the present invention. belongs to

Claims (20)

In electronic devices,
a memory that stores one or more instructions; and
a processor to execute the one or more instructions stored in the memory, the processor executing the one or more instructions to:
obtaining an avatar mode based on at least one of user status information and content characteristics output from the electronic device;
Determine the degree of personification of the avatar based on the content characteristics,
Obtain attributes of the avatar based on the avatar mode and the degree of personification,
An electronic device that controls rendering of the avatar according to the acquired attributes of the avatar. According to claim 1,
The processor, by executing the one or more instructions,
The electronic device that obtains the user's state information by sensing at least one of the user's body, posture, eye tracking, and voice characteristics to determine the user's emotion or intention. According to claim 1,
The processor, by executing the one or more instructions,
The avatar mode includes at least one of a companion avatar mode, an assistant avatar mode, and a feature avatar mode;
Acquiring one of the companion avatar mode, the assistant avatar mode, and the feature avatar mode as the avatar mode based on at least one of the user status information and the content characteristics. According to claim 3,
The processor, by executing the one or more instructions,
and determining the avatar mode as the companion avatar mode when the state information of the user represents the user's emotional characteristics. According to claim 3,
The processor, by executing the one or more instructions,
and determining the avatar mode as the assistant avatar mode when the state information of the user indicates a state in which the user needs assistance. According to claim 3,
The processor, by executing the one or more instructions,
and determining the avatar mode as a feature avatar mode based on the characteristics of the content when the status information of the user indicates neutral. According to claim 1,
The processor, by executing the one or more instructions,
Determine the degree of personification of the avatar based on the content characteristics,
An electronic device that determines the strength of the personification of the avatar to be higher or lower according to the content characteristics. According to claim 1,
The attributes of the avatar include at least one of a character type of the avatar, an appearance of the character, a behavior of the character, an animation property performed on the character, or a location where the character is rendered. According to claim 1,
The processor, by executing the one or more instructions,
The electronic device that controls the rendering of the avatar to be performed on a display of the electronic device or controls the rendering of the avatar to be performed on an auxiliary device connected through communication. According to claim 9,
The auxiliary device includes a head mounted display,
The processor, by executing the one or more instructions,
An electronic device that controls the avatar to be rendered in three dimensions (3D) on the head mounted display. In the operating method of the electronic device,
Obtaining an avatar mode based on at least one of user status information and content characteristics output from the electronic device;
An operation of determining the degree of personification of the avatar based on the content characteristics;
Obtaining attributes of the avatar based on the avatar mode and the degree of personification, and
and controlling rendering of the avatar according to the acquired attributes of the avatar. According to claim 11,
The method of operating an electronic device further comprising obtaining state information of the user by sensing at least one of the user's body, posture, eye tracking, and voice characteristics to determine the user's emotion or intention. According to claim 11,
The avatar mode includes at least one of a companion avatar mode, an assistant avatar mode, and a feature avatar mode;
The method,
and acquiring one of the companion avatar mode, the assistant avatar mode, and the feature avatar mode as the avatar mode based on at least one of the user status information and the content characteristics. . According to claim 13,
and determining the avatar mode as the companion avatar mode when the state information of the user indicates the user's emotional characteristics. According to claim 13,
and determining the avatar mode as the assistant avatar mode when the state information of the user indicates a state in which the user needs help. According to claim 13,
The method of operating the electronic device further comprising determining the avatar mode as a feature avatar mode based on the characteristics of the content when the status information of the user indicates neutral. According to claim 11,
An operation of determining the degree of personification of the avatar based on the content characteristics, and
The method of operating the electronic device further comprising an operation of determining higher or lower level of personification of the avatar according to the content characteristics. According to claim 11,
The attribute of the avatar includes at least one of a character type of the avatar, an appearance of the character, a behavior of the character, an animation property performed on the character, or a location where the character is rendered. According to claim 11,
The method of operating the electronic device further comprising controlling rendering of the avatar to be performed on a display of the electronic device or controlling rendering of the avatar to be performed on an auxiliary device connected through communication. A computer-readable recording medium in which a program for performing a method of operating an electronic device is recorded, the method of operating the electronic device comprising:
Obtaining an avatar mode based on at least one of user status information and content characteristics output from the electronic device;
An operation of determining the degree of personification of the avatar based on the content characteristics;
Obtaining attributes of the avatar based on the avatar mode and the degree of personification, and
and controlling rendering of the avatar according to the acquired attributes of the avatar.

Images