KR20240019666A - Method and electronic device for displaying avatar in augmented reality - Google Patents

Abstract

According to various embodiments, an electronic device may include a communication module, a display module, a memory, and a processor operatively connected to the communication module, the display module, and the memory. The processor may display the first conversation window with the first other party in the virtual space through the display module. The processor may check the size of the first avatar corresponding to the first counterpart in response to the first user input moving from the first point to the second point in relation to the first chat window. Based on the confirmed size of the first avatar, the processor may check the area of the horizontal plane where the first avatar can be located and whether the first avatar can be placed in a three-dimensional space based on the area. The processor converts the first chat window into the first avatar in response to the possible placement of the first avatar when the second point according to the first user input is included in the area of the identified horizontal plane and a three-dimensional space based on the area. You can. The processor may display the first avatar based on the area of the confirmed horizontal plane and a three-dimensional space based on the area. Various other embodiments may be possible.

Description

METHOD AND ELECTRONIC DEVICE FOR DISPLAYING AVATAR IN AUGMENTED REALITY}

The descriptions below relate to methods and electronic devices for displaying avatars in augmented reality (AR).

With the recent development of technology, electronic devices are moving away from the uniform rectangular shape and are gradually being transformed into various shapes. For example, in order to increase user convenience, electronic devices are gradually developing into wearable electronic devices that can be worn on any part of the body.

Wearable electronic devices may include head mounted display (HMD) devices that can be worn on the head, such as glasses. For example, a wearable electronic device may include glasses-type augmented reality (AR) glasses and/or smart glasses that implement various contents on transparent glass (eg, lenses). For another example, a wearable electronic device may be an HMD device and may include a video see-through (VST) device that captures a real environment using a camera and displays the captured image in a form that overlaps a virtual image. Wearable electronic devices, HMD devices, and/or VST devices may provide virtual reality services and/or augmented reality services (eg, augmented reality worlds, augmented reality functions) to users using cameras. For example, with the HMD device worn on the user's head, the HMD device may implement virtual reality and/or augmented reality in response to execution of an augmented reality-related application on a communication-connected electronic device, and provide the user with a virtual reality and/or augmented reality. Reality services and/or augmented reality services may be provided.

The above information may be provided as background technology for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may apply as prior art with respect to the present disclosure.

According to one embodiment, an electronic device uses an HMD device to provide an augmented reality service based on augmented reality (e.g., virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR) technology. (e.g. virtual reality service) can be provided to users. The electronic device may display a conversation window containing a conversation message with another electronic device (e.g., another user using another electronic device, a conversation partner) within the augmented reality area.

According to one embodiment, within an augmented reality area, a wearable electronic device that can display a 3D (dimension) avatar (e.g., object) created in response to a conversation partner and a conversation message included in a conversation window. can be provided.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

According to one embodiment, the electronic device may include a communication module, a display module, a memory, and a processor operatively connected to the communication module, the display module, and the memory. The processor may display the first conversation window with the first other party in the virtual space through the display module. The processor may check the size of the first avatar corresponding to the first counterpart in response to the first user input moving from the first point to the second point in relation to the first chat window. Based on the confirmed size of the first avatar, the processor may check the area of the horizontal plane where the first avatar can be located and whether the first avatar can be placed in a three-dimensional space based on the area. The processor converts the first chat window into the first avatar in response to the possible placement of the first avatar when the second point according to the first user input is included in the area of the identified horizontal plane and a three-dimensional space based on the area. You can. The processor may display the first avatar based on the area of the confirmed horizontal plane and a three-dimensional space based on the area.

In a method of displaying an avatar according to an embodiment, a first chat window with a first party can be displayed in a virtual space. The method according to one embodiment may check the size of the first avatar corresponding to the first other person in response to the first user input moving from the first point to the second point in relation to the first chat window. The method according to one embodiment can check, based on the confirmed size of the first avatar, the area of the horizontal plane where the first avatar can be located and whether it is possible to place the first avatar in a three-dimensional space based on the area. The method according to one embodiment provides a first chat window in response to possible placement of the first avatar when the second point according to the first user input is included in the area of the identified horizontal plane and the three-dimensional space based on the area. 1 Can be converted to an avatar. The method according to one embodiment may display the first avatar based on the area of the identified horizontal plane and a three-dimensional space based on the area.

According to one embodiment, a non-transitory computer-readable storage medium (or computer program product) storing one or more programs may be described. According to one embodiment, one or more programs, when executed by a processor of an electronic device, display a first chat window with a first party in a virtual space, and, in relation to the first chat window, display a second chat window at a first point. In response to a first user input moving to a point, an operation of confirming the size of a first avatar corresponding to the first other party, based on the confirmed size of the first avatar, an area of a horizontal plane in which the first avatar can be located, and An operation of checking whether it is possible to place a first avatar in a three-dimensional space based on an area, when a second point according to a first user input is included in the area of the confirmed horizontal plane and a three-dimensional space based on the area, the first Converting the first chat window to the first avatar in response to the avatar being available for placement, and displaying the first avatar based on the area of the identified horizontal plane and a three-dimensional space based on the area.

According to one embodiment, a wearable electronic device (eg, HMD electronic device) may provide an environment in which a user can naturally exchange messages with a conversation partner within augmented reality (eg, virtual reality). The wearable electronic device may display a 3D (dimension) avatar corresponding to the conversation partner in an augmented reality space, while simultaneously displaying a conversation message with the conversation partner (eg, avatar).

In one embodiment, a wearable electronic device may implement a conversation situation (eg, a communication situation) such that a user is actually having a conversation with a conversation partner within an augmented reality space. According to one embodiment, when implementing a chat window as an avatar, the wearable electronic device checks the area of the horizontal plane where the avatar can be located based on the size of the avatar and whether the avatar can be placed in a three-dimensional space based on the area. If the avatar can be arranged, the chat window can be converted to an avatar and displayed. According to one embodiment, the wearable electronic device can display avatars corresponding to a plurality of conversation partners in an augmented reality space, create a group conversation window based on the plurality of avatars, and avatars participating in the group conversation window and A conversation situation can be implemented. According to one embodiment, the user's sense of immersion according to the augmented reality situation may be improved.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.
1 is a block diagram of an electronic device in a network environment according to embodiments of the present disclosure.
Figure 2 is a perspective view schematically showing the configuration of a wearable electronic device according to an embodiment of the present disclosure.
3 is a block diagram of an electronic device according to an embodiment of the present disclosure.
Figure 4 is a flowchart illustrating a method for creating an avatar in augmented reality according to an embodiment of the present disclosure.
Figure 5 is a flowchart illustrating a method of generating a plurality of avatars and creating a group chat window in which the plurality of avatars participate in augmented reality according to an embodiment of the present disclosure.
FIG. 6A is an exemplary diagram illustrating a situation in which a first chat window is displayed in augmented reality according to an embodiment of the present disclosure.
FIG. 6B is an example diagram illustrating an area in which a first avatar corresponding to a first conversation partner participating in a first conversation window can be displayed in augmented reality according to an embodiment of the present disclosure.
FIG. 6C is an example diagram illustrating a situation in which the first avatar can be displayed in augmented reality according to an embodiment of the present disclosure.
FIG. 6D is an example diagram illustrating a situation in which the first avatar and the first chat window are displayed together in augmented reality according to an embodiment of the present disclosure.
FIG. 7A is an example diagram illustrating a situation in which a second chat window is displayed in augmented reality where a first avatar is displayed according to an embodiment of the present disclosure.
FIG. 7B is an example diagram illustrating an area where a second avatar corresponding to a second conversation partner participating in a second conversation window can be displayed in augmented reality where a first avatar is being displayed according to an embodiment of the present disclosure.
FIG. 7C is an example diagram illustrating a situation in which a second avatar and a second chat window are displayed together in augmented reality where a first avatar is being displayed according to an embodiment of the present disclosure.
FIG. 7D is an example diagram illustrating a situation in which a group chat window is created based on the first avatar and the second avatar in augmented reality where the first avatar and the second avatar are displayed according to an embodiment of the present disclosure.
FIG. 7E is an example diagram illustrating a situation in which a conversation message is displayed in a group chat window in which a first conversation partner and a second conversation partner are participating according to an embodiment of the present disclosure.
FIG. 8A is an example diagram illustrating a situation in which a user input occurs to move a first avatar in augmented reality according to an embodiment of the present disclosure.
FIG. 8B is an example diagram illustrating a situation in which a first chat window in which a first conversation partner is participating is displayed when the first avatar is unable to move according to an embodiment of the present disclosure.
FIG. 9 is an example diagram illustrating a situation in which an avatar with a relatively higher priority is displayed with emphasis when a first avatar and a second avatar are displayed together in augmented reality according to an embodiment of the present disclosure.
FIG. 10 is an example diagram illustrating a situation in which emotional information is displayed using the first avatar when a conversation message containing emotional information is received in augmented reality where the first avatar is displayed according to an embodiment of the present disclosure. .
Figure 11 is an example diagram illustrating a situation in which a plurality of avatars corresponding to a plurality of users participating in a group chat window are displayed according to an embodiment of the present disclosure.
Figure 12 is an example diagram illustrating a situation in which a conversation message being displayed is scrolled in augmented reality according to an embodiment of the present disclosure.
Figure 13 is an example diagram recommending that the latest message and group message be converted to an avatar in a chat window according to an embodiment of the present disclosure.
Figure 14 is an example of displaying in color a group chat window in which an avatar displayed in augmented reality according to an embodiment of the present disclosure is participating.
Figure 15 is an example of determining the size of an avatar differently based on the size of the space in which the avatar can move according to an embodiment of the present disclosure.
FIG. 16A is an example of implementing the shape of an avatar differently based on the size of the space in which the avatar can move according to an embodiment of the present disclosure.
FIG. 16B is an example diagram illustrating a situation in which the shape of the avatar changes based on the size of the space in which the avatar can move when the avatar moves according to an embodiment of the present disclosure.
FIG. 16C is an example diagram illustrating a situation in which the shape of an avatar changes in response to a gesture input input to the avatar according to an embodiment of the present disclosure.
FIG. 17A is an example diagram illustrating a situation in which a first chat window is displayed in augmented reality based on a portable electronic device according to an embodiment of the present disclosure.
FIG. 17B is an example diagram illustrating a situation in which a first avatar corresponding to a first conversation partner participating in a first conversation window is displayed in augmented reality based on a portable electronic device according to an embodiment of the present disclosure.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself, where artificial intelligence is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support high frequency bands (eg, mmWave bands), for example, to achieve high data rates. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external electronic devices 102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Figure 2 is a perspective view schematically showing the configuration of a wearable electronic device according to an embodiment of the present disclosure.

The wearable electronic device 200 shown in FIG. 2 (e.g., the electronic device 101 in FIG. 1) is at least partially similar to the electronic device 101 in FIG. 1, or further includes other embodiments of the electronic device 101. can do. The wearable electronic device 200 may include glasses-type augmented reality (AR) glasses, smart glasses, and/or a video see-through (VST) device.

Referring to FIG. 2, the wearable electronic device 200 includes a bridge 201, a first rim 210, a second rim 220, a first end piece 230, and a second rim 201. It may include an end piece 240, a first temple (temple, 250), and/or a second temple (260).

According to one embodiment, the bridge 201 may connect the first limb 210 and the second limb 220. The bridge 201 may be positioned above the user's nose when the user wears the wearable electronic device 200. The bridge 201 may separate the first limb 210 and the second limb 220 based on the user's nose.

According to one embodiment, the bridge 201 may include a camera module 203, a first eye tracking camera 205, a second eye tracking camera 207, and/or an audio module 209.

According to one embodiment, the camera module 203 (e.g., the camera module 180 in FIG. 1) photographs the front (e.g., -y-axis direction) of the user (e.g., the user of the wearable electronic device 200). and image data can be obtained. The camera module 203 may capture an image corresponding to the user's field of view (FoV) or measure the distance to a subject (eg, an object). The camera module 203 may include an RGB camera, a high resolution (HR) camera, and/or a photo video (PV) camera. The camera module 203 may include a color camera that performs an auto focus (AF) function and an optical image stabilization (OIS) function to obtain a high-quality image.

According to one embodiment, the first gaze tracking camera 205 and the second gaze tracking camera 207 may track the user's gaze. The first gaze tracking camera 205 and the second gaze tracking camera 207 may capture the user's eyes looking in a direction opposite to the shooting direction of the camera module 203. For example, the first eye tracking camera 205 may partially photograph the user's left eye, and the second eye tracking camera 207 may partially photograph the user's right eye. The first eye tracking camera 205 and the second eye tracking camera 207 may detect the user's pupils (eg, left eye and right eye) and track the gaze direction corresponding to the detected pupil movement. The tracked gaze direction can be used to move the center of a virtual image including a virtual object in response to the gaze direction. The first eye tracking camera 205 and/or the second eye tracking camera 207 may be, for example, an EOG sensor (electro-oculography or electrooculogram), a coil system, a dual Purkinje system, bright pupil systems or dark pupil systems. At least one method can be used to track the user's gaze.

According to one embodiment, the audio module 209 (eg, the audio module 170 in FIG. 1) may be placed between the first eye tracking camera 205 and the second eye tracking camera 207. The audio module 209 may convert the user's voice into an electrical signal or convert an electrical signal into sound (e.g., sound source data, audio signal). Audio module 209 may include a microphone.

According to one embodiment, the first rim 210 and the second rim 220 may form a frame (eg, an eyeglasses frame) of the wearable electronic device 200 (eg, AR glasses). The first rim 210 may be disposed in a first direction (eg, x-axis direction) of the bridge 201. The first rim 210 may be placed in a position corresponding to the user's left eye. The second rim 220 may be disposed in a second direction (eg, -x-axis direction) of the bridge 201, which is opposite to the first direction (eg, x-axis direction). The second rim 220 may be placed in a position corresponding to the user's right eye. The first rim 210 and the second rim 220 may be made of metal and/or a non-conductive material (eg, polymer).

According to one embodiment, the first rim 210 may surround and support at least a portion of the first glass 215 (eg, first display) disposed on the inner peripheral surface. The first glass 215 may be positioned in front of the user's left eye. The second rim 220 may surround and support at least a portion of the second glass 225 (eg, a second display) disposed on the inner peripheral surface. The second glass 225 may be positioned in front of the user's right eye. A user of the wearable electronic device 200 can view the foreground (eg, actual image) of an external object (eg, subject) through the first glass 215 and the second glass 225. The wearable electronic device 200 can implement augmented reality by displaying a virtual image overlaid on the foreground (eg, real image) of an external object. According to one embodiment, the wearable electronic device 200 may implement virtual reality or express virtual reality through the first glass 215 and the second glass 225.

According to one embodiment, the first glass 215 and the second glass 225 may include a projection type transparent display. The first glass 215 and the second glass 225 are each transparent plates (or transparent screens) and can form a reflective surface, and the image generated by the wearable electronic device 200 is reflected (e.g., reflected) through the reflective surface. It may be total internal reflection and enter the user's left and right eyes. In one embodiment, the first glass 215 may include an optical waveguide that transmits light generated from a light source of the wearable electronic device 200 to the user's left eye. For example, the optical waveguide may be formed of glass, plastic, or polymer material, and may include a nanopattern (e.g., a polygonal or curved grating structure) formed on the inside or surface of the first glass 215. or mesh structure). The optical waveguide may include at least one of at least one diffractive element (eg, a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (eg, a reflective mirror). The optical waveguide may guide display light emitted from the light source to the user's eyes using at least one diffractive element or reflective element included in the optical waveguide. In one embodiment, the diffractive element may include an input/output optical member, and the reflective element may include a total internal reflection (TIR). For example, light emitted from a light source may be guided to an optical waveguide through an input optical member, and light traveling inside the optical waveguide may be guided toward the user's eyes through an output optical member. The second glass 225 may be implemented in substantially the same way as the first glass 215 .

According to one embodiment, the first glass 215 and the second glass 225 are, for example, a liquid crystal display (LCD), a digital mirror device (DMD), or silicon. It may include a liquid crystal on silicon (LCoS) device, an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the first glass 215 and the second glass 225 are made of one of a liquid crystal display device, a digital mirror display device, or a silicon liquid crystal display device, the wearable electronic device 200 includes the first glass 215 and the second glass 225. It may include a light source that radiates light to the screen output area of the second glass 225. In another embodiment, when the first glass 215 and the second glass 225 are capable of generating light on their own, for example, when they are made of one of organic light emitting diodes or micro LEDs, the wearable electronic device 200 can provide a virtual image of good quality to the user even if it does not include a separate light source.

According to one embodiment, the first rim 210 may include a first microphone 211, a first recognition camera 213, a first light emitting device 217, and/or a first display module 219. there is. The second rim 220 may include a second microphone 221, a second recognition camera 223, a second light emitting device 227, and/or a second display module 229.

In one embodiment, the first light-emitting device 217 and the first display module 219 may be included in the first end piece 230, and the second light-emitting device 227 and the second display module 229 may be included in the first end piece 230. It may be included in the second end piece 240.

According to one embodiment, the first microphone 211 and/or the second microphone 221 are capable of receiving the voice (e.g., speech voice, speech sound source data) of the user wearing the wearable electronic device 200 on the head. and the received user's voice can be converted into an electrical signal.

According to one embodiment, the first recognition camera 213 and/or the second recognition camera 223 may recognize the surrounding space of the wearable electronic device 200. The first recognition camera 213 and/or the second recognition camera 223 may detect a user's gesture within a certain distance (eg, a certain space) of the wearable electronic device 200. The first recognition camera 213 and/or the second recognition camera 223 uses GS (GS) in which the rolling shutter (RS) phenomenon can be reduced in order to detect and track the user's quick hand movements and/or fine movements of the fingers. may include a global shutter) camera. The wearable electronic device 200 uses the first eye tracking camera 205, the second eye tracking camera 207, the first recognition camera 213, and/or the second recognition camera 223 to detect the user's left eye and /Or, among the right eyes, the eye corresponding to the main eye and/or the auxiliary eye can be detected. For example, the wearable electronic device 200 may detect the eye corresponding to the primary eye and/or the secondary eye based on the user's gaze direction with respect to an external object or virtual object.

According to one embodiment, the first light-emitting device 217 and/or the second light-emitting device 227 include a camera module 203, a first eye tracking camera 205, a second eye tracking camera 207, and a second eye tracking camera 207. It may emit light to increase the accuracy of the first recognition camera 213 and/or the second recognition camera 223. The first light-emitting device 217 and/or the second light-emitting device 227 are used to increase accuracy when photographing the user's eyes using the first eye tracking camera 205 and/or the second eye tracking camera 207. It can be used as an auxiliary tool. When the first light-emitting device 217 and/or the second light-emitting device 227 captures a user's gesture using the first recognition camera 213 and/or the second recognition camera 223, the first light-emitting device 217 and/or the second light-emitting device 227 may be used in a dark environment or in various places. It can be used as an auxiliary means when it is not easy to detect an object (e.g., a subject) to be photographed due to mixing of light sources and reflected light. The first light emitting device 217 and/or the second light emitting device 227 may include, for example, at least one of an LED, an IR LED, and/or a xenon lamp.

According to one embodiment, the first display module 219 and/or the second display module 229 emit light and use the first glass 215 and/or the second glass 225 to display the user's left eye and /Or it can be delivered to the right eye. The first glass 215 and/or the second glass 225 may display various image information using light emitted through the first display module 219 and/or the second display module 229. The first display module 219 and/or the second display module 229 may include the display module 160 of FIG. 1 . The wearable electronic device 200 displays a foreground for an external object (e.g., a real external environment) and an image emitted through the first display module 219 and/or the second display module 229 (e.g., a virtual An object (a 3-dimension (3D) object) may be displayed by overlapping it through the first glass 215 and/or the second glass 225 .

According to one embodiment, the first end piece 230 may be coupled to a portion (eg, x-axis direction) of the first rim 210. The second end piece 240 may be coupled to a portion (eg, -x-axis direction) of the second rim 220. In one embodiment, the first light emitting device 217 and the first display module 219 may be included in the first end piece 230. The second light emitting device 227 and the second display module 229 may be included in the second end piece 240.

According to one embodiment, the first end piece 230 may connect the first rim 210 and the first temple 250. The second end piece 240 may connect the second rim 220 and the second temple 260.

According to one embodiment, the first temple 250 may be operatively connected to the first end piece 230 using the first hinge portion 255. The first hinge portion 255 may be rotatable so that the first temple 250 is folded or unfolded with respect to the first rim 210 . The first temple 250 may extend, for example, along the left side of the user's head. The distal portion (e.g., in the y-axis direction) of the first temple 250 may be bent to be supported by, for example, the user's left ear when the user wears the wearable electronic device 200. The second temple 260 may be operatively connected to the second end piece 240 using the second hinge portion 265. The second hinge portion 265 may be rotatable so that the second temple 260 is folded or unfolded with respect to the second rim 220 . The second temple 260 may extend, for example, along the right side of the user's head. The distal portion (e.g., in the y-axis direction) of the second temple 260 may be bent to be supported by, for example, the user's right ear when the user wears the wearable electronic device 200.

According to one embodiment, the first temple 250 includes a first printed circuit board 251, a first sound output module 253 (e.g., the sound output module 155 of FIG. 1), and/or a first battery. 257 (e.g., battery 189 in FIG. 1). The second temple 260 includes a second printed circuit board 261, a second sound output module 263 (e.g., the sound output module 155 in FIG. 1), and/or a second battery 267 (e.g., FIG. It may include a battery (189) of 1.

According to one embodiment, the first printed circuit board 251 and/or the second printed circuit board 261 include the processor 120, memory 130, interface 177, and/or wireless communication disclosed in FIG. 1. Various electronic components such as the module 192 (eg, at least some of the components included in the electronic device 101 of FIG. 1) may be disposed. The processor may include, for example, one or more of a central processing unit, an application processor, a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor. The first printed circuit board 251 and/or the second printed circuit board 261 may include, for example, a printed circuit board (PCB), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB). there is. In some embodiments, the first printed circuit board 251 and/or the second printed circuit board 261 include a main PCB, a slave PCB partially overlapping with the main PCB, and/or between the main PCB and the slave PCB. It may include an interposer substrate. The first printed circuit board 251 and/or the second printed circuit board 261 are connected to other components (e.g., the camera module 203, the first eye tracking camera) using an electrical path such as an FPCB and/or a cable. (205), second eye tracking camera 207, audio module 209, first microphone 211, first recognition camera 213, first light emitting device 217, first display module 219, Second microphone 221, second recognition camera 223, second light emitting device 227, second display module 229, first sound output module 253 and/or second sound output module 263 ) can be electrically or operationally connected to. For example, FPCB and/or cables may be placed on at least a portion of the first limb 210, bridge 201, and/or second limb 220. In some embodiments, the wearable electronic device 200 may include only one of the first printed circuit board 251 or the second printed circuit board 261.

According to one embodiment, the first sound output module 253 and/or the second sound output module 263 may transmit an audio signal to the user's left and/or right ears. The first sound output module 253 and/or the second sound output module 263 may include, for example, a piezo speaker (eg, bone conduction speaker) that transmits an audio signal without a speaker hole. In some embodiments, the wearable electronic device 200 may include only one of the first audio output module 253 or the second audio output module 263.

According to one embodiment, the first battery 257 and/or the second battery 267 uses a power management module (e.g., the power management module 188 of FIG. 1) to manage the first printed circuit board 251. ) and/or power may be supplied to the second printed circuit board 261. The first battery 257 and/or the second battery 267 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In some embodiments, the wearable electronic device 200 may include only one of the first battery 257 or the second battery 267.

According to one embodiment, the wearable electronic device 200 may include a sensor module (eg, sensor module 176 in FIG. 1). The sensor module may generate an electrical signal or data value corresponding to the internal operating state of the wearable electronic device 200 or the external environmental state. Sensor modules include, for example, gesture sensors, gyro sensors, barometric pressure sensors, magnetic sensors, acceleration sensors, grip sensors, color sensors, IR (infrared) sensors, biometric sensors (e.g. HRM sensors), temperature sensors, humidity sensors, Alternatively, it may further include at least one of an illumination sensor. In some embodiments, the sensor module may be configured to include various biometric sensors (or biometric sensors), such as an olfactory sensor (e-nose sensor), an electromyography sensor (EMG sensor), an electroencephalogram sensor (EEG sensor), an electrocardiogram sensor (ECG sensor), or an iris sensor. The user's biometric information can be recognized using a recognition sensor.

According to one embodiment, in the above description, the wearable electronic device 200 has been described as a device that displays augmented reality using the first glass 215 and the second glass 225, but it is not limited to this and the virtual It may be a device that displays reality (virtual reality; VR).

3 is a block diagram of an electronic device according to an embodiment of the present disclosure.

The electronic device 101 of FIG. 3 may be at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or may further include other embodiments of the electronic device 101. . The electronic device 101 of FIG. 3 may be a diagram illustrating some configurations of the wearable electronic device 200 disclosed in FIG. 2 .

Referring to FIG. 3, an electronic device (e.g., the electronic device 101 of FIG. 1 or the wearable electronic device 200 of FIG. 2) includes a processor (e.g., the processor 120 of FIG. 1) and a memory (e.g., the electronic device 101 of FIG. 1). may include a memory 130), a display module (e.g., the display module 160 of FIG. 1), and/or a communication module (e.g., the communication module 190 of FIG. 1). The memory 130 may store an augmented reality-related application 311 for implementing an augmented reality space and a message application 312 for sending and receiving conversation messages with a conversation partner. According to one embodiment, the memory 130 may store information related to the creation of an avatar in order to implement and display the avatar in an augmented reality space.

According to one embodiment, the processor 120 of the electronic device 101 runs a program (e.g., the program 140 of FIG. 1, the augmented reality-related application 311, and the message application 312) stored in the memory 130. By executing, it can control at least one other component (e.g., hardware and/or software component) and perform various data processing or operations. According to one embodiment, processor 120 is operatively, functionally, and/or electrically connected to memory 130, display module 160, and/or communication module 190. Can be connected (electrically).

According to one embodiment, the processor 120 can execute an augmented reality-related application 311 and implement an augmented reality space based on the augmented reality-related application 311. The processor 120 may display an augmented reality space through the display module 160. According to one embodiment, the processor 120 may automatically execute the augmented reality-related application 311 in response to a situation in which the electronic device 101 is worn on the user's head and provide an augmented reality service to the user. You can.

According to one embodiment, the electronic device 101 may be operatively connected to a dedicated controller for providing an augmented reality service to a user within an augmented reality space. For example, the dedicated controller may include input means for at least partially controlling the augmented reality-related application 311. For example, a dedicated controller may include a hardware-implemented input device (e.g., a control device, remote control, stylus pen), or a software-implemented (e.g., visually implemented) input module within the augmented reality space, and May include input means. For another example, the electronic device 101 may recognize the user's hand (e.g., finger) as a dedicated controller, and may control the augmented reality-related application 311 based on input and movement using the user's hand. there is. According to one embodiment, the processor 120 can detect user input (e.g., touch input, gesture input) and movement based on a dedicated controller, and functions related to augmented reality and /Or an action can be performed.

According to one embodiment, the processor 120 may execute various functions and programs related to augmented reality within the augmented reality space. For example, the processor 120 may execute the message application 312 in the augmented reality space and display a conversation window (e.g., a message window) based on the message application 312 through the display module 160. there is. According to one embodiment, the processor 120 may display various functions related to augmented reality as icons included in the menu bar, and may check user input (e.g., gesture input, touch input) corresponding to the icon. The processor 120 may execute functions and programs corresponding to icons in response to user input.

According to one embodiment, the display module 160 may visually display programs running in the electronic device 101 and activated functions. For example, when executing the augmented reality-related application 311, the processor 120 can implement an augmented reality space and display the augmented reality space through the display module 160. According to one embodiment, the electronic device 101 may visually display an augmented reality space implemented based on an augmented reality service to the user through the display module 160. In one embodiment, a user wearing the electronic device 101 on their head may feel as if they are actually living in the real world.

According to one embodiment, the communication module 190 may perform a communication connection with an external electronic device (eg, the electronic devices 102 and 104 of FIG. 1). For example, the electronic device 101 (e.g., a wearable electronic device) and the external electronic devices 102 and 104 (e.g., a portable electronic device) communicate through various communication methods (e.g., a wired communication channel) through the communication module 190. , a wireless communication channel), communication may be connected.

According to one embodiment, the electronic device 101 can independently implement an augmented reality space and provide an augmented reality service to a user. According to one embodiment, the electronic device 101 is operatively connected to the external electronic device 102 (e.g., a portable electronic device, a smartphone) through the communication module 190. It may be controlled at least partially by . For example, when the augmented reality space has already been implemented in the external electronic device 102, the electronic device 101 may only perform an operation of displaying the implemented augmented reality space through the display module 160.

According to one embodiment, the communication module 190 includes a first sub-communication module supporting a communication connection based on the first network 198 of FIG. 1 and a communication connection based on the second network 199 of FIG. 1. It may include a second sub-communication module. For example, the electronic device 101 (e.g., HMD device, wearable electronic device, VST device) may be connected to communicate with the external electronic device 102 (e.g., portable electronic device, smartphone) through the first sub-communication module. and can be connected to another electronic device (e.g., the electronic device of a conversation partner) through the second sub-communication module. The electronic device 101 is in a communication connection (e.g., BT communication connection) with the external electronic device 102, and is in a communication connection (e.g., sending or receiving a conversation message) with another electronic device (e.g., the electronic device of a conversation partner). ) can be performed.

According to one embodiment, the processor 120 of the electronic device 101 (e.g., a wearable electronic device) may detect a situation in which the electronic device 101 is worn on the user's head and may detect the augmentation stored in the memory 130. A reality-related application (311) can be executed. The processor 120 can implement an augmented reality space based on the executed augmented reality-related application 311, and display the implemented augmented reality space through the display module 160. The processor 120 can execute the message application 312 stored in the memory 130 while the augmented reality space is implemented, and display a chat window corresponding to one area of the augmented reality space. For example, within the augmented reality space, a chat window may be displayed corresponding to the direction the user is gazing, or a chat window may be displayed corresponding to a predetermined area. The chat window can be displayed in the form of a 3D (3-dimension) object displayed in harmony with the augmented reality space. In one embodiment, the processor 120 may detect a user touching or gripping the chat window (eg, a gesture) and check whether a user input has occurred in the chat window. For example, the processor 120 may be operatively connected to a dedicated controller for controlling the augmented reality-related application 311, and may detect a user's touch input and grip input based on the dedicated controller. For example, a chat window may be moved from a first point to a second point in response to user input within the augmented reality space.

In one embodiment, the electronic device 101 may be operatively connected to a dedicated controller for providing augmented reality services within an augmented reality space. For example, the dedicated controller may include input means for at least partially controlling the augmented reality-related application 311. For example, a dedicated controller may include a hardware-implemented input device (e.g., controller, remote control, input pen, stylus pen), or a software-implemented (e.g., visually implemented) input device within the augmented reality space. It may include an input module and an input means. For another example, the electronic device 101 may recognize a part of the user's body (e.g., hand, finger) as a dedicated controller, and run the augmented reality-related application 311 based on input and movement using the user's hand. You can control it. According to one embodiment, the processor 120 can detect user input (e.g., touch input, gesture input) and movement based on a dedicated controller, and functions related to augmented reality and /Or an action can be performed. The processor 120 may perform various functions and/or operations related to augmented reality using a dedicated controller.

According to one embodiment, the processor 120 may convert a chat window into an avatar and display it in an augmented reality space. For example, an avatar may include a 3D characterized object. The processor 120 can create an avatar corresponding to a conversation partner participating in the conversation window and display it in the augmented reality space. According to one embodiment, the processor 120 may check the area in which the avatar can be placed (e.g., the area of the horizontal plane in which the avatar can be placed and the three-dimensional space based on the area) in response to the user input to the chat window. . For example, the processor 120 may check the area corresponding to the horizontal plane and the area of the area in the augmented reality space. For example, a horizontal plane may mean a plane perpendicular to the direction of gravity. The processor 120 may recognize a horizontal plane and an area included within an angle and range that can be recognized as a substantially horizontal plane as a horizontal plane. According to one embodiment, the processor 120 moves the user input related to the chat window from the first point to the second point when the user input stays in the placeable area of the avatar (e.g., the second point) for a set time. , the chat window can be converted into an avatar. The processor 120 may display an avatar (eg, an avatar corresponding to a conversation partner) in the augmented reality space instead of a conversation window.

According to one embodiment, the processor 120 may implement multiple layers based on the augmented reality space and perform augmented reality services according to each layer. The processor 120 can implement an augmented reality space corresponding to an actual external environment and display multiple layers within the augmented reality space.

For example, the first layer may include a group window for executing an application related to an augmented reality service. The group window may include widget icons corresponding to applications related to the augmented reality service. For example, the first layer may be implemented based on an area within the augmented reality space that is a set distance away from the user. The first layer may include a vertical layer in augmented reality space. According to one embodiment, the processor 120 may display a chat window included in a group window based on the first layer and detect a user input to the chat window. The processor 120 may change the color of the chat window in response to a user input to the chat window. The processor 120 may reflect and display a highlight effect to distinguish the object from the surrounding environment based on the detected user input object.

For example, the second layer may include a workstation area for implementing applications related to augmented reality services in various forms (eg, avatar form). The processor 120 may visually display the workstation area based on the second layer. Based on the second layer, the processor 120 may identify a plurality of horizontal areas (eg, floor area) where the avatar can be located. According to one embodiment, the processor 120 may check whether the generated avatar is a locationable area based on the second layer.

For example, the third layer may include a menu bar for executing an application related to augmented reality. The processor 120 may display the implemented menu bar based on the third layer. The third layer may include a menu bar composed of multiple icons (eg, application icons for executing applications).

According to one embodiment, the processor 120 may divide the augmented reality space into a plurality of layers, and may separately detect user input corresponding to each layer. The processor 120 can perform an augmented reality service in an augmented reality space based on multiple layers. According to one embodiment, the processor 120 may check a user input for a chat window displayed based on the first layer and check a user input for dragging the chat window into the augmented reality space. The processor 120 responds to a user input (e.g., a user input related to a chat window moving to a specific area and specific space in the augmented reality space) and, based on the second layer, creates an avatar (e.g., an avatar into which the chat window has been converted). ) can be identified. According to one embodiment. Processor 120 is based on the second layer. The area of the horizontal plane where the avatar can be located and the three-dimensional space based on the area can be confirmed, and the chat window can be converted into at least one avatar and displayed.

According to one embodiment, the first layer, the second layer, and/or the third layer may be displayed at least partially overlapping each other. The processor 120 can distinguish user input according to each layer.

According to one embodiment, the electronic device 101 may display an avatar corresponding to a conversation partner in an augmented reality space, and provide the user with an experience like actually having a conversation with a conversation partner in the augmented reality space (e.g., augmented reality service). ) can be provided.

According to one embodiment, an electronic device (e.g., the electronic device 101 of FIGS. 1 and/or 3 and/or the electronic device 200 of FIG. 2) includes a communication module (e.g., the electronic device 101 of FIG. 1 and/or FIG. 3). communication module 190), a display module (e.g., display module 160 in FIGS. 1 and/or 3), a memory (e.g., memory 130 in FIG. 1 and/or 3), and a communication module ( 190), a display module 160, and a processor (eg, processor 120 of FIGS. 1 and/or 3) operatively connected to the memory 130. The processor 120 may display the first chat window 620 with the first other party 601 within the virtual space 630 through the display module 160. In relation to the first chat window 620, the processor 120 determines the size of the first avatar 602 corresponding to the first other party 601 in response to the first user input moving from the first point to the second point. You can check. Based on the confirmed size of the first avatar 602, the processor 120 creates an area 603 of the horizontal plane where the first avatar 602 can be located and a first avatar (603) in a three-dimensional space 660 based on the area. 602) can be checked whether the arrangement is possible. The processor 120 determines that placement of the first avatar 602 is possible when the second point according to the first user input is included in the area 603 of the confirmed horizontal plane and the three-dimensional space 660 based on the area. In response, the first chat window 620 can be converted into the first avatar 602. The processor 120 may display the first avatar 602 based on the identified horizontal plane area 603 and a three-dimensional space 660 based on the area.

According to one embodiment, when the first chat window 620 is located in the confirmed horizontal plane area 603 and the three-dimensional space 660 based on the area, the processor 120 determines the horizontal plane area 603 and 3 The shape of the first avatar 602 corresponding to the dimensional space 660 can be confirmed. The processor 120 may convert the first chat window 620 into the first avatar 602 based on the confirmed shape of the first avatar 602.

According to one embodiment, the processor 120 may display the second chat window 720 with the second counterpart 701 while the first avatar 602 is being displayed in the virtual space 630. The processor 120 determines the size of the second avatar 702 corresponding to the second counterpart 701 in response to the second user input moving from the third point to the fourth point in relation to the second chat window 720. You can check. Based on the confirmed size of the second avatar 702, the processor 120 may check the area 703 of the horizontal plane where the second avatar 702 can be located and a three-dimensional space based on the area. When the fourth point according to the second user input is included in the area 703 of the confirmed horizontal plane and the three-dimensional space based on the area, the processor 120 converts the second chat window 720 to the second avatar 702. It can be converted. The processor 120 may display the second avatar 702 based on the identified horizontal plane area 703 and a three-dimensional space based on the area.

According to one embodiment, the processor 120 may check the distance between the first avatar 602 and the second avatar 702 within the virtual space 630. If the confirmed distance is less than or equal to a set threshold, the processor 120 may create a group chat window in which the first avatar 602 and the second avatar 702 participate.

According to one embodiment, when creating a group chat window, the processor 120 may integrate the first area where the first avatar 602 is located and the second area where the second avatar 702 is located into one. The processor 120 may display at least one conversation message included in the group conversation window within the virtual space 630.

According to one embodiment, the first color of the first area is determined based on the address book category containing the first counterpart 601, and the second area is determined by the first color based on the address book category containing the second counterpart 701. Two colors are determined, and the third area integrated based on the first and second areas may be displayed in one color.

According to one embodiment, the processor 120 may check a user input for separating the first avatar 602 and the second avatar 702 integrated into the third area. The processor 120 may divide the third area into a first area of the first avatar 602 and a second area of the second avatar 702 in response to the confirmed user input. The processor 120 may individually display the first chat message included in the first chat window of the first avatar 602 and the second chat message included in the second chat window of the second avatar 702.

According to one embodiment, the area 603 of the horizontal surface where the first avatar 602 can be positioned may be determined based on the horizontal and vertical lengths calculated corresponding to the size of the first avatar 602. The area-based three-dimensional space 660 may be determined based on the determined horizontal length, determined vertical length, and height of the first avatar 602.

According to one embodiment, the processor 120 maintains the location of the first chat window 620 moving by user input for a set time in the area 603 of the horizontal plane in which the location is confirmed and the three-dimensional space 660 based on the area. In this case, the first chat window 620 may be converted to the first avatar 602 in response to the availability of the first avatar 602. If the position of the first chat window 620 moving by user input is not maintained for the time set in the confirmed area 603 of the horizontal plane and the three-dimensional space 660 based on the area, the processor 120 The chat window 620 can be maintained.

According to one embodiment, when the first avatar 602 and the second avatar 702 are displayed at least partially overlapping within the virtual space 630, the processor 120 displays the first chat window 620 and the second avatar 620. You can check the priority of the chat window 720. The processor 120 can reflect the highlight effect and display it on the avatar corresponding to the chat window with relatively high priority.

According to one embodiment, when the first other party 601 receives a conversation message containing emotional information, the processor 120 creates an animation effect so that the first avatar 602 displays emotional information based on the emotional information. It can be displayed.

According to one embodiment, the processor 120 may display a group chat window in which the first counterpart 601 and the second counterpart 701 are participating within the virtual space 630. The processor 120 determines the size of the group in which the first avatar 602 and the second avatar 702 are combined in response to a third user input moving from the fifth point to the sixth point in relation to the group chat window. You can. Based on the confirmed size of the group, the processor 120 may check the area of the horizontal plane where the first avatar 602 and the second avatar 702 can be located and a three-dimensional space based on the area. If the sixth point according to the third user input is included in the area of the confirmed horizontal plane and the three-dimensional space based on the area, the processor 120 divides the group chat window into the first avatar 602 and the second avatar 702. It can be converted. The processor 120 may display the first avatar 602 and the second avatar 702 based on the area of the confirmed horizontal plane and a three-dimensional space based on the area.

According to one embodiment, the processor 120 may determine the size and appearance of the first avatar 602 based on information related to the first other party 601, and determine the size and appearance of the first avatar 602 based on information related to the second other party 701. The size and appearance of the second avatar 702 can be determined.

Figure 4 is a flowchart illustrating a method for creating an avatar in augmented reality according to an embodiment of the present disclosure.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

The electronic device 101 of FIG. 4 may be at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or may further include other embodiments of the electronic device 101. .

According to one embodiment, the processor (e.g., processor 120 of FIG. 3) of the electronic device 101 (e.g., a wearable electronic device) is configured to enable the electronic device 101 to be worn on a part of the user's body (e.g., the head). The situation can be detected, and an augmented reality-related application (e.g., the augmented reality-related application 311 of FIG. 3) stored in the memory (e.g., the memory 130 of FIGS. 1 and 3) can be executed. The processor 120 may implement an augmented reality space based on the executed augmented reality-related application 311, and display the implemented augmented reality space through a display module (e.g., the display module 160 in FIGS. 1 and 3). It can be displayed through . The processor 120 may execute a message application (e.g., the message application 312 of FIG. 3) stored in the memory 130 while the augmented reality space is implemented, and display a chat window corresponding to one area of the augmented reality space. can do.

In operation 401, the processor of the electronic device 101 (eg, processor 120 of FIG. 3) may display a first chat window with the first counterpart within the augmented reality space. For example, when an icon for executing the message application 312 is included in the augmented reality space, the processor 120 may execute the message application 312 in response to a user input corresponding to the icon. For another example, when a first message written by a first party is received, the processor 120 may display a notification message for the first message, and in response to a user input corresponding to the notification message, augmented reality The first chat window with the first other party can be displayed in the space.

According to one embodiment, the processor 120 may implement multiple layers based on the augmented reality space and perform augmented reality services according to each layer. For example, the processor 120 can implement an augmented reality space corresponding to an actual external environment and display multiple layers within the augmented reality space. For example, the first layer may include a group window for executing an application related to an augmented reality service. The group window may include a first conversation window with the first other party. The first layer may include a vertical layer in augmented reality space. The processor 120 may detect a user input (eg, touch input, drag input) for the first chat window based on the first layer.

In operation 403, the processor 120 may check the size of the avatar corresponding to the first counterpart in response to a user input related to the displayed first chat window. The user input may include a drag input that moves from a first point to a second point within the augmented reality space. For example, the size of the avatar corresponding to the first counterpart may be a size set as a default value, or may include a size arbitrarily set by the user. According to one embodiment, the appearance of the avatar may be determined based on information related to the first counterpart (e.g., gender, age, height, weight, physical characteristics, image information). For example, the size and appearance of the avatar corresponding to the first counterpart may be modified or changed according to the user's settings.

In operation 405, the processor 120 may check the area of the horizontal plane where the avatar can be located within the augmented reality space and the three-dimensional space based on the area, based on the size of the avatar. According to one embodiment, when implementing an augmented reality space, the processor 120 may check the area corresponding to the horizontal plane and the area of the area. For example, the processor 120 may check a user input moving from a first point to a second point, and determine the area of the horizontal plane corresponding to the second point and a three-dimensional space (e.g., three-dimensional area) based on the area. )can confirm. The processor 120 may check an area in the augmented reality space where an avatar of a specific size can be placed (e.g., the area of the horizontal plane where the avatar can be placed and a three-dimensional space based on the area). For example, the processor 120 may display different colors and shades so that the area where the avatar can be placed is visually distinguished from other areas. For example, the second layer may include a workstation area for implementing applications related to augmented reality services in various forms (eg, avatar form). The processor 120 may visually display the workstation area based on the second layer. The processor 120 may display an area where the avatar can be placed based on the second layer. For example, the processor 120 may check the area corresponding to the second point according to the user input based on the second layer and determine whether an avatar can be placed in the confirmed area.

According to another embodiment, if there is no placeable area for the avatar in the augmented reality space implemented based on the user's field of view (e.g., gaze direction), the processor 120 may operate in an augmented reality space other than the user's field of view. You can check the area where the avatar can be placed. The processor 120 may display guide information (eg, user interface) that guides the user's gaze so that the user can check the possible placement area.

In operation 407, the processor 120 determines a three-dimensional space (e.g., an avatar location) in which a user input that has moved relative to the first chat window (e.g., a user input that moves the first chat window from a first point to a second point) has been identified. You can check whether it is located within a three-dimensional space for a certain period of time. For example, an action in which a second point is fixed (e.g., maintained) for a certain period of time in a three-dimensional space according to a user input (e.g., an input that drags the first chat window) is when the user intentionally moves the first chat window to the first avatar. It can include the action you want to convert to . If the user input is not maintained within the 3D space for a certain period of time, the user may intend to move the first chat window to another location.

For another example, when the user input moves without being located in the confirmed three-dimensional space, the processor 120 returns the first dialog window to the position before the user input was confirmed, or returns it to the point where the user input ends. The first chat window can be displayed. For example, the operation of continuously moving the first chat window may include the user intentionally trying to move the first chat window from the first position to the second position. The processor 120 may move the first chat window from the first position to the second position based on the second layer.

If the user input that selects the first chat window in operation 407 remains located in the confirmed three-dimensional space (e.g., a space in which an avatar can be located) for a certain period of time, the processor 120 determines the location of the user input in operation 409 ( For example: coordinates), the first chat window can be converted into the first avatar corresponding to the first other person. For example, processor 120 may display a preview image of the first avatar before converting to the first avatar if user input to the first chat window is maintained (e.g., the first chat window is clicked). And, when the user input to the first chat window is completed (e.g., the user input to the first chat window is canceled), the preview image of the first avatar may be changed to the first avatar and displayed. For example, the preview image of the first avatar may include an image displaying a blurry image of the first avatar, or an image displaying an afterimage of the first avatar.

If the user input for selecting the first chat window in operation 407 is not maintained for a certain period of time while being located in the confirmed three-dimensional space (e.g., a space where the avatar can be located), the processor 120 selects the first chat window in operation 413. The displayed state can be maintained as is. For example, in a situation where the first chat window is continuously moving, the processor 120 may display the first chat window as is.

After displaying the first avatar in operation 409, the processor 120 may display the first avatar and the conversation message included in the first chat window in the augmented reality space in operation 411. For example, the processor 120 may display a conversation message in the form of a speech bubble of the first avatar, rather than in the form of a conversation window. For example, as if having an actual conversation with the first avatar in an augmented reality space, the processor 120 may display a conversation message in the form of a speech bubble while moving the mouth of the first avatar.

According to one embodiment, the processor 120 may recognize an object placed in an augmented reality space, and when implementing an avatar, may display the avatar so that the avatar performs an action related to the object. For example, the processor 120 may check a sofa placed in the augmented reality space, and when the user input for selecting the first chat window is located on the sofa, the first avatar is shown sitting on the sofa, and the first avatar is shown sitting on the sofa. 1 The avatar can be displayed. For another example, the processor 120 may check exercise equipment placed in the augmented reality space, and when the user input that selects the first chat window is located on the exercise equipment, the first avatar exercises using the exercise equipment. You can display what you are doing. According to one embodiment, the electronic device 101 can set actions and situations that can be implemented within the augmented reality space, and when implementing an avatar, it can reflect animation effects on the avatar based on the set actions and situations. , the avatar can be displayed.

According to one embodiment, the electronic device 101 may implement a conversation situation (eg, a communication situation) such that the user of the electronic device 101 is having an actual conversation with a conversation partner within the augmented reality space. According to one embodiment, the processor 120 of the electronic device 101 may display a first avatar corresponding to the first conversation partner in the augmented reality space, and a body part (e.g., mouth) of the first avatar may be displayed in the augmented reality space. Augmented reality services such as actual conversation while moving can be provided to users. According to one embodiment, the user's sense of immersion in the augmented reality service may be improved.

Figure 5 is a flowchart illustrating a method of generating a plurality of avatars and creating a group chat window in which the plurality of avatars participate in augmented reality according to an embodiment of the present disclosure.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

The electronic device 101 of FIG. 5 may be at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or may further include other embodiments of the electronic device 101. .

According to one embodiment, the processor (e.g., processor 120 of FIG. 3) of the electronic device 101 (e.g., a wearable electronic device) is configured to enable the electronic device 101 to be worn on a part of the user's body (e.g., the head). The situation can be detected, and an augmented reality-related application (e.g., the augmented reality-related application 311 of FIG. 3) stored in the memory (e.g., the memory 130 of FIGS. 1 and 3) can be executed. The processor 120 may implement an augmented reality space based on the executed augmented reality-related application 311, and display the implemented augmented reality space through a display module (e.g., the display module 160 in FIGS. 1 and 3). It can be displayed through . The processor 120 may execute a message application (e.g., the message application 312 of FIG. 3) stored in the memory 130 while the augmented reality space is implemented, and display a chat window corresponding to one area of the augmented reality space. can do.

In operation 501, the processor of the electronic device 101 (e.g., the processor 120 of FIG. 3) may create a first avatar based on the first other party participating in the first chat window within the augmented reality space. For example, the processor 120 may perform operations 401 to 409 of FIG. 4 to create a first avatar.

In operation 503, the processor 120 may display the created first avatar in the augmented reality space. For example, the processor 120 may display the first avatar based on the area in which the first chat window is located (e.g., 3D space, 3D area). Based on the size of the first avatar, the processor 120 may check the area of the horizontal plane where the avatar can be located and the three-dimensional space based on the area. For example, the processor 120 may move the position of the first avatar in response to a user's drag input corresponding to the first avatar. According to one embodiment, the processor 120 may determine the appearance of the first avatar corresponding to the first counterpart stored in the address book, based on the user's settings and information related to the first counterpart. For example, processor 120 may process the request based on information related to the first party (e.g., gender, age, height, weight, skin color (e.g., white race, black race, yellow race), and additional information stored in memory 130). 1 You can decide the appearance of your avatar. In one embodiment, the first avatar in the augmented reality space may be displayed corresponding to the first counterpart.

In operation 505, the processor 120 may create a second avatar based on the second party participating in the second chat window in the augmented reality space. For example, the processor 120 may perform operations 401 to 409 of FIG. 4 to create a second avatar. According to one embodiment, while the first avatar is displayed in the augmented reality space, the processor 120 may display a second chat window based on a messaging application (eg, the messaging application 312 of FIG. 3). For example, the second chat window may be in a state in which the second party is participating. The processor 120 may create a second avatar corresponding to the second counterpart.

In operation 507, the processor 120 may display the generated second avatar in the augmented reality space. For example, while the first avatar is displayed in the augmented reality space, the processor 120 may display the second avatar based on the area (e.g., 3D space, 3D area) where the second chat window is located. Based on the size of the second avatar, the processor 120 may check the area of the horizontal plane where the avatar can be located and the three-dimensional space based on the area. According to one embodiment, the processor 120 may determine the appearance of the second avatar based on the user's settings and information related to the second counterpart. For example, the processor 120 may process the request based on information related to the second party (e.g., gender, age, height, weight, skin color (e.g., white race, black race, yellow race), and additional information stored in memory 130). 2 You can decide the appearance of the avatar. In one embodiment, the processor 120 may display a first avatar corresponding to the first counterpart and a second avatar corresponding to the second counterpart in the augmented reality space.

In operation 509, the processor 120 may check whether the distance between the first avatar and the second avatar is less than or equal to a set threshold. For example, the positions of the first avatar and the second avatar may be moved in response to a user input (eg, a drag input). The processor 120 can check the distance between the first avatar and the second avatar and check whether the first avatar and the second avatar are located closer or further away than a set threshold. For example, the processor 120 may check the distance between the first area (eg, floor area) where the first avatar is located and the second area (eg, floor area) where the second avatar is located. For example, the processor 120 may check the distance between the first line corresponding to the edge of the first area and the second line corresponding to the edge of the second area.

When the distance between the first avatar and the second avatar in operation 509 is less than a set threshold, the processor 120 may create a group chat window in which the first avatar and the second avatar participate together in operation 511. For example, the processor 120 may send a plurality of users (e.g., a first counterpart corresponding to the first avatar, a second counterpart corresponding to the second avatar, and a user of the electronic device 101) based on the message application 312. ) can create a group chat window in which people participate. For example, the processor 120 may combine the first area where the first avatar is located and the second area where the second avatar is located and display them as one area. The processor 120 may group the first avatar and the second avatar. The area displayed below the grouped first avatar and the second avatar (e.g., an area where the first area and the second area are combined) may be displayed in a color different from the color of the first area of the first avatar.

In operation 513, the processor 120 may display the first avatar, the second avatar, and the conversation message included in the group conversation window in the augmented reality space. For example, the processor 120 may display a conversation message in the form of a speech bubble, such as when a first avatar and a second avatar are talking to each other. According to one embodiment, the processor 120 may implement a situation in which the first avatar and the second avatar express emotions while facing each other based on the conversation message.

According to one embodiment, the processor 120 may check a user input (e.g., gesture input) for separating the grouped first avatar and the second avatar, and display the first avatar and the second avatar visually apart from each other. You can. For example, the user input may include a first input for moving the first avatar in a first direction and a second input for moving the second avatar in a second direction opposite to the first direction. The first input and the second input may include a double touch input that occurs substantially at the same time. When the first avatar and the second avatar are separated, the processor 120 controls the conversation message included in the group chat window not to be displayed in the augmented reality space, while controlling the first conversation message and the second avatar corresponding to the first avatar. The second conversation message can be displayed individually. When the first avatar and the second avatar are separated, the processor 120 divides the area displayed below the first avatar and the second avatar (e.g., the area where the first area and the second area are combined) into the first area and the second avatar. It can be displayed separately into areas. The first area and the second area may be displayed in different colors.

FIG. 6A is an exemplary diagram illustrating a situation in which a first chat window is displayed in augmented reality according to an embodiment of the present disclosure. FIG. 6B is an example diagram illustrating an area in which a first avatar corresponding to a first conversation partner participating in a first conversation window can be displayed in augmented reality according to an embodiment of the present disclosure. FIG. 6C is an example diagram illustrating a situation in which the first avatar can be displayed in augmented reality according to an embodiment of the present disclosure. FIG. 6D is an example diagram illustrating a situation in which the first avatar and the first chat window are displayed together in augmented reality according to an embodiment of the present disclosure.

The electronic device 101 of FIGS. 6A to 6D is at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or further includes other embodiments of the electronic device 101. can do.

According to one embodiment, the processor (e.g., processor 120 of FIG. 3) of the electronic device 101 (e.g., a wearable electronic device) is configured to enable the electronic device 101 to be worn on a part of the user's body (e.g., the head). The situation can be detected, and an augmented reality-related application (e.g., the augmented reality-related application 311 of FIG. 3) stored in the memory (e.g., the memory 130 of FIGS. 1 and 3) can be executed. The processor 120 may implement an augmented reality space 630 based on the executed augmented reality-related application 311, and the implemented augmented reality space 630 may be displayed on a display module (e.g., in FIGS. 1 and 3 It can be displayed through the display module 160). A user wearing the electronic device 101 on his head can check the augmented reality space 630 corresponding to the user's viewing angle.

Referring to FIG. 6A, the processor 120 may display a menu bar 610 consisting of a plurality of icons for executing various functions and programs related to augmented reality within the augmented reality space 630. For example, the menu bar 610 may be displayed in an at least partially overlapping (eg, overlay) form based on a partial area of the augmented reality space 630. The menu bar 610 includes a gallery icon 611 for displaying at least one image stored in memory (e.g., memory 130 in FIG. 3) in the augmented reality space 630, and an address book icon for displaying address book information ( 612), a message icon 613 for displaying a conversation message, an alarm icon 614 for setting an alarm, a calendar icon 615 for displaying a calendar, and augmentation for implementing augmented reality (AR). It may include a reality icon 616 and/or a virtual reality icon 617 for implementing virtual reality (VR). According to one embodiment, icons included in the menu bar 610 can be changed according to the user's settings, and are not limited to the icons described above. According to one embodiment, the processor 120 can check user input or gesture input occurring in the augmented reality space 630, and can distinguish and recognize user input corresponding to each icon.

Referring to FIG. 6A, the processor 120 may execute a message application (e.g., the message application 312 of FIG. 3) stored in the memory 130 while the augmented reality space 630 is implemented. The first chat window 620 in which the first other party 601 is participating may be displayed corresponding to one area of . For example, the first counterpart 601 may refer to a conversation partner who is sending and receiving messages through the first chat window 620 based on the message application 312. The first other party 601 may be displayed as the first avatar. The first chat window 620 may include at least one chat message 621.

According to one embodiment, when the electronic device 101 receives a conversation message 621 from the first counterpart 601, the electronic device 101 may display a notification message in one area of the augmented reality space 630. The processor 120 may check a user input for selecting the notification message, and in response to confirmation of the user input, may display a first chat window 620 in which the first other party 601 is participating.

According to one embodiment, the processor 120 may check the size of the avatar corresponding to the first other party 601 in response to a user input related to the first chat window 620. For example, the size of the first avatar 602 corresponding to the first counterpart 601 may be a size set as a default value, or may include a size arbitrarily set by the user. According to one embodiment, the size and appearance of the first avatar 602 may be determined based on information (e.g., gender, age, height, weight, physical characteristics, image information) related to the first counterpart 601. . For example, the first avatar 602 may be modified or changed according to the user's settings.

Referring to FIG. 6B, based on the size of the first avatar 602, the processor 120 calculates an area 650 of a horizontal plane in which the first avatar 602 can be located within the augmented reality space 630 and an area in the area. You can check the three-dimensional space based on it. According to one embodiment, the processor 120 responds to a user input 640 related to the first chat window 620 (e.g., a drag input to move the first chat window 620), and determines the area of the horizontal plane where the avatar can be positioned. You can check (650). For example, the processor 120 may display the area 650 of the horizontal plane where the avatar can be located within the augmented reality space 630 by reflecting a highlight effect to visually distinguish it from the surrounding background.

Referring to FIG. 6C, the first area 603 (e.g., floor area) where the first avatar 602 is located is based on the size of the first avatar 602 (e.g., torso circumference of the first avatar 602). can be decided. The first area 603 may be implemented as an area that can include the first avatar 602 in the first area 603 when the first avatar 602 is viewed from above. According to one embodiment, the three-dimensional space 660 in which the first avatar 602 can be located has the horizontal length, vertical length, and length set corresponding to the size of the first avatar 602. It may include a three-dimensional space 660 in the form of a rectangular parallelepiped formed based on height. For example, the three-dimensional space 660 may be determined based on the size of the first area 603 and the height (eg, height) of the first avatar 602. The volume of the three-dimensional space 660 can be implemented as the size of the avatar included in the space. According to one embodiment, the processor 120 provides a gesture input to stretch or shrink the implemented avatar (e.g., a first input that grips at least two points and stretches outward or inward while at least two points are gripped). You can check the second input (e.g., drag input) to reduce the size, and increase or decrease the size of the implemented avatar.

According to one embodiment, the three-dimensional space 660 is a three-dimensional user interface that appears visually when a set gesture input (e.g., a gesture input to change the size of the avatar, a gesture input to move the avatar) is confirmed. You can. For example, the three-dimensional space 660 may disappear from being visually visible when a set time elapses. According to one embodiment, the three-dimensional space 660 may be displayed with a highlight effect reflected according to specific conditions. For example, when the avatar moves, if the avatar is unable to move, the color of the three-dimensional space 660 may change to a different color. When the avatar moves, even if the shape of the avatar changes, the color of the 3D space 660 may change to another color.

6B and 6C, the processor 120, in response to a user input related to the first chat window 620, determines an area 650 of a horizontal plane in which an avatar can be located within the augmented reality space 630 and an area 650 in the area. You can check the three-dimensional space based on it. The processor 120 may check the user input 640 (eg, drag input) for moving the first chat window 620. For example, the processor 120 may reflect and display a highlight effect based on the movement path (e.g., movement trajectory) of the user input 640. The processor 120 may check whether the user input related to the first chat window 620 is located within a confirmed three-dimensional space (eg, a space in which an avatar can be located) for a certain period of time. When the user input related to the first chat window 620 is located within the confirmed three-dimensional space for a certain period of time, the processor 120 converts the first chat window 620 into a first avatar 602, as shown in FIG. 6C. can do. For example, the processor 120 may display an animation effect that changes from the form of the first chat window 620 to the form of the first avatar 602.

According to one embodiment, in the operation of converting the first chat window 620 into the first avatar 602, when the user input to the first chat window 620 is maintained (e.g., the first chat window 620 (while the touch input is maintained) the processor 120 displays a preview image of the first avatar 602 (e.g., a blurred image of the first avatar 602, or an afterimage of the first avatar 602) one image) can be displayed. For example, the preview image of the first avatar 602 may include an image in which the first avatar 602 is displayed as a dotted line and an image in which the first avatar 602 is displayed in a blurry manner. In the operation of converting the first chat window 620 into the first avatar 602, when the user input to the first chat window 620 is completed (e.g., the touch input to the first chat window 620 is released) state) the processor 120 may display the first avatar 602.

Referring to FIG. 6D, the processor 120 may display the first avatar 602 and the first chat message 604 included in the first chat window 620. For example, the first conversation message 604 may include a conversation message transmitted and received with the first other party 601 through the first conversation window 620.

According to one embodiment, the processor 120 confirms the area 650 of the horizontal plane where the avatar can be located and a three-dimensional space based on the area, and the first avatar 602 can be located within the confirmed three-dimensional space. If present, the first avatar 602 may be displayed in the augmented reality space 630. The first avatar 602 may be displayed on the first area 603 (eg, an area on the floor where the first avatar 602 is located). According to one embodiment, the first area 603 may be displayed in various colors, and the color may be determined based on the type of the first chat window 602 in which the first avatar is participating (e.g., independent chat window, group chat window). You can.

FIG. 7A is an example diagram illustrating a situation in which a second chat window is displayed in augmented reality where a first avatar is displayed according to an embodiment of the present disclosure. FIG. 7B is an example diagram illustrating an area where a second avatar corresponding to a second conversation partner participating in a second conversation window can be displayed in augmented reality where a first avatar is being displayed according to an embodiment of the present disclosure. FIG. 7C is an example diagram illustrating a situation in which a second avatar and a second chat window are displayed together in augmented reality where a first avatar is being displayed according to an embodiment of the present disclosure. FIG. 7D is an example diagram illustrating a situation in which a group chat window is created based on the first avatar and the second avatar in augmented reality where the first avatar and the second avatar are displayed according to an embodiment of the present disclosure. FIG. 7E is an example diagram illustrating a situation in which a conversation message is displayed in a group chat window in which a first conversation partner and a second conversation partner are participating according to an embodiment of the present disclosure.

The electronic device 101 of FIGS. 7A to 7E is at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or further includes other embodiments of the electronic device 101. can do.

Referring to FIG. 7A, with the first avatar 602 displayed in the augmented reality space 630 according to FIG. 6D, the processor 120 of the electronic device 101 runs a message application (e.g., the message application of FIG. 3). A second chat window 720 based on 312)) may be displayed. For example, when the first avatar 602 is being displayed and a message 721 is received from the second counterpart 701, the processor 120 may display a notification message indicating receipt of the message 721. You can. The processor 120 may display a second chat window 720 in the augmented reality space 630 in response to a user input for the notification message. For example, the second counterpart 701 may refer to a conversation partner who is sending and receiving messages through the second chat window 720 based on the message application 312.

Referring to FIG. 7B, the processor 120 responds to the user input 740 related to the second chat window 720, checks the size of the second avatar 702 corresponding to the second counterpart 701, and confirms the size of the second avatar 702. Based on the size of the second avatar 702, the area 750 of the horizontal plane where the second avatar 702 can be positioned within the augmented reality space 630 and the three-dimensional space based on the area can be confirmed. For example, the size of the second avatar 702 corresponding to the second counterpart 701 may be a size set as a default value, or may include a size arbitrarily set by the user. According to one embodiment, the size and appearance of the second avatar 702 may be determined based on information (e.g., gender, age, height, weight, physical characteristics, image information) related to the second counterpart 701. . The second avatar 702 in the augmented reality space may be displayed corresponding to the second counterpart 701.

According to one embodiment, the processor 120 responds to a user input 740 related to the second chat window 720 (e.g., a drag input to move the second chat window 720), and determines the area of the horizontal plane where the avatar can be positioned. You can check (750). For example, the processor 120 may visually display the area 750 of the horizontal plane where the avatar can be located within the augmented reality space 630. According to one embodiment, the processor 120 confirms the area 750 of the horizontal plane where the second avatar 702 can be located and a three-dimensional space based on the area, and the confirmed three-dimensional space is the second avatar 702. ), it can be determined that the area 750 of the corresponding horizontal plane allows the second avatar 702 to be located.

Referring to FIG. 7C, the processor 120 may display the second avatar 702 in the augmented reality space 630 and also display the second chat message 704 included in the second chat window 720. . For example, the second conversation message 704 may include a conversation message transmitted and received with the second counterpart 701 through the second conversation window 720. According to one embodiment, the first avatar 602 corresponding to the first counterpart 601 and the second avatar 702 corresponding to the second counterpart 701 may be displayed together in the augmented reality space 630. . The first area 603 where the first avatar 602 is located and the second area 703 where the second avatar 702 is located may be displayed in different colors. For example, the area of the first area 603 may be determined based on the size of the first avatar 602, and the area of the second area 703 may be determined based on the size of the second avatar 702. there is. The first conversation message 604 with the first avatar 602 and the second conversation message 704 with the second avatar 702 may be displayed in different colors and appear in different spatial phases in the augmented reality space 630. It can be displayed in . According to one embodiment, the positions of the first avatar 602 and the second avatar 702 may be fixed in the augmented reality space 630, but in response to a change in the user's position, the first avatar 602 and the second avatar 702 may be fixed in the augmented reality space 630. 2 The gaze direction of the avatar 702 may be changed. For example, the first avatar 602 and the second avatar 702 may be displayed continuously looking at the user. According to one embodiment, as the user's location changes, one of the first avatar 602 and the second avatar 702 may be located relatively further away, and the avatar located further away may be displayed relatively small. .

Referring to FIG. 7D, the first avatar 602 and the second avatar 702 may be moved by a user's input (eg, drag input). According to one embodiment, when the distance between the first avatar 602 and the second avatar 702 is less than a set threshold, the processor 120 allows the first avatar 602 and the second avatar 702 to participate together. You can create a group chat window. For example, when the distance between the first avatar 602 and the second avatar 702 is less than or equal to a set threshold, the first area 603 of the first avatar 602 and the second area of the second avatar 702 703 may be displayed as a third area 710 combined into one. According to one embodiment, the processor 120 may display each chat window in different colors. For example, the processor 120 may set the color of each chat window to be different based on the category of the address book (e.g., family, friends, coworkers, relatives). The processor 120 may determine the color of the area where the avatar is located (e.g., the first area 603, the second area 703, and the third area 710) based on the color set in each chat window. For example, when the first avatar 602 and the second avatar 702 are placed apart from each other, the processor 120 may display the first area 603 of the first avatar 602 in the first color. , the second area 703 of the second avatar 702 may be displayed in a second color. When the first avatar 602 and the second avatar 702 participate in one group chat window, the processor 120 may display the third area 710 in a third color. The processor 120 may display different colors in response to each chat window.

Referring to FIG. 7E, the processor 120 generates a first avatar 602, a second avatar 702, and a conversation message ( 711) can be displayed together in the augmented reality space 630.

FIG. 8A is an example diagram illustrating a situation in which a user input occurs to move a first avatar in augmented reality according to an embodiment of the present disclosure. FIG. 8B is an example diagram illustrating a situation in which a first chat window in which a first conversation partner is participating is displayed when the first avatar is unable to move according to an embodiment of the present disclosure.

The electronic device 101 of FIGS. 8A to 8B is at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or further includes other embodiments of the electronic device 101. can do.

Referring to FIG. 8A, with the first avatar 602 displayed in the augmented reality space 630 according to FIG. 6D, the processor 120 confirms the user input 840 for moving the first avatar 602. And, a situation in which the first avatar 602 is moving can be displayed. In response to the user input 840 for the first avatar 602, the processor 120 may check the area 850 of the horizontal plane where the first avatar 602 can be located and a three-dimensional space based on the area. According to one embodiment, when the first avatar 602 is located in a space where it is difficult to locate in the augmented reality space 630, the processor 120 changes the first area 603 of the first avatar 602 to a set color. It can be changed to (e.g. red, a color that indicates that the avatar is difficult to position). The processor 120 can continuously check the state in which the first avatar 602 is moving, and can continuously change the color of the first area 603 based on the position of the first avatar 602.

Referring to FIG. 8B, the processor 120 can confirm that the user input 840 has been completed while the first avatar 602 has moved to a space where it is difficult to locate, and displays the first avatar 602 in the first chat window ( It can be displayed by changing it to 820). According to one embodiment, the processor 120 operates under conditions in which the avatar is not displayed (e.g., the area 850 of the horizontal plane where the first avatar 602 can be located, and not the three-dimensional space based on the area, but the remainder of the space other than the area 850). If the condition of moving into space) is met, the first avatar 602 can be changed to the first chat window 820. For example, the first chat window 820 may be displayed based on the point at which the user input 840 is completed. According to one embodiment, the processor 120 may display the first chat window 820 based on the first layer. For example, the processor 120 may display a group window containing widget icons corresponding to a plurality of applications based on the first layer, and the group window 820 may be included in the group window. A window can be displayed.

FIG. 9 is an example diagram illustrating a situation in which an avatar with a relatively higher priority is displayed with emphasis when a first avatar and a second avatar are displayed together in augmented reality according to an embodiment of the present disclosure.

Referring to FIG. 9, when the first avatar 602 and the second avatar 702 are displayed together in the augmented reality space 630 according to FIG. 7C, the processor 120 processes the first avatar 602 The priority of the conversation message 622 and the second conversation message 722 of the second avatar 702 can be confirmed. For example, the latest message may have a relatively higher priority than other messages. Referring to FIG. 9, in a situation where the latest message is received from the second counterpart 701 corresponding to the second avatar 702, the second avatar 702 and the second conversation message 722 are sent to the first avatar 602. ) and may be displayed with relative emphasis compared to the first conversation message 622. For example, the processor 120 displays the second avatar 702 and the second conversation message 722 with a highlight effect, or blurs the first avatar 602 and the first conversation message 622. It can be displayed. For example, the processor 120 displays the font and speech bubble of the second conversation message 722 thickly or changes the color so that the second conversation message 722 is displayed relatively more clearly than the first conversation message 622. can be displayed differently.

FIG. 10 is an example diagram illustrating a situation in which emotional information is displayed using the first avatar when a conversation message containing emotional information is received in augmented reality where the first avatar is displayed according to an embodiment of the present disclosure. .

Referring to FIG. 10, in a situation where the first avatar 1002 corresponding to the first counterpart 601 is displayed in the augmented reality space 630, the processor 120 displays emotional or behavioral information (e.g., joy, sadness, You can receive a conversation message (e.g., an emoticon 1001) containing happiness, surprise, embarrassment, walking, running, etc., and perform a gesture reflecting the emotion or behavioral information based on the first avatar 1002. can do. For example, when an emoticon 1001 containing an emotion of sadness is received, the processor 120 uses a gesture set to the facial expression and movement of the first avatar 1002 so that the first avatar 1002 displays the emotion of sadness. It can be displayed to reflect (e.g. a gesture showing sadness). For example, when an animation effect for expressing a specific emotion is stored in the memory 130, the processor 120 can reflect the animation effect and display it on the first avatar 1002.

Figure 11 is an example diagram illustrating a situation in which a plurality of avatars corresponding to a plurality of users participating in a group chat window are displayed according to an embodiment of the present disclosure.

The electronic device 101 of FIG. 11 may be at least partially similar to the electronic device 101 of FIG. 1 and/or the wearable electronic device 200 of FIG. 2, or may further include other embodiments of the electronic device 101. .

Referring to FIG. 11 , the processor 120 may display a group chat window 1120 in which the first counterpart 601 and the second counterpart 701 are participating in the augmented reality space 630. For example, when a new chat message is received in the group chat window 1120, the processor 120 may display a notification message notifying the receipt of a new chat message. The processor 120 may display a group chat window 1120 in the augmented reality space 630 in response to a user input for the notification message.

Referring to FIG. 11, the processor 120 responds to a user input 1140 related to the group chat window 1120, to the first avatar 602 and the second counterpart 701 corresponding to the first counterpart 601. The size of the corresponding second avatar 702 is confirmed, and based on the confirmed sizes of the first avatar 602 and the second avatar 702, the first avatar 602 is located within the augmented reality space 630. ) and the second avatar 702 can be located together on the area of the horizontal plane and the three-dimensional space based on the area. For example, the processor 120 may integrate the three-dimensional space corresponding to the first avatar 602 and the three-dimensional space corresponding to the second avatar 702 into one, and the integrated space satisfies 3 You can check the dimensional space. According to one embodiment, when the integrated space for the first avatar 602 and the second avatar 702 is included in a three-dimensional space within the augmented reality space 630, the processor 120 processes the first avatar 602 ) and the second avatar 702 can be displayed in a combined form. For example, the first area where the first avatar 602 is located and the second area where the second avatar 702 is located may be integrated into one and displayed as a third area 710. The processor 120 may display the first avatar 602 and the second avatar 702 in combination and display the group conversation message included in the group conversation window 1120 in the augmented reality space 630.

According to one embodiment, while the first avatar 602 and the second avatar 702 are combined, the processor 120 performs a user input ( Example: gesture input) can be checked. For example, the user input may include a first input for moving the first avatar 602 in a first direction and a second input for moving the second avatar 702 in a second direction opposite to the first direction. may include. The first input and the second input may include a double touch input that occurs substantially at the same time. For another example, the user input may include only one of a first input and a second input. The processor 120 may check the first input to the first avatar 602 and, in response to the first input, move only the first avatar 602 to the first position corresponding to the first input. . The processor 120 may check the second input for the second avatar 702 and, in response to the second input, may move only the second avatar 702 to a second location corresponding to the second input. .

According to one embodiment, the processor 120, in response to a user input that separates the first avatar 602 and the second avatar 702, separates the integrated third area 710 from the first avatar 602. It can be divided into a first area and a second area of the second avatar 702 and displayed. The processor 120 may display the first area of the first avatar 602 and the second area of the second avatar 702 in different colors. The processor 120 controls the group chat message not to be displayed while displaying the first chat message included in the first chat window in which the first avatar 602 is participating and the second chat message included in the second chat window in which the second avatar 702 is participating. The second conversation message can be displayed individually. For example, when the first avatar 602 and the second avatar 702 are separated, the processor 120 may display a first conversation message on the head of the first avatar 602 and the second avatar ( A second conversation message can be displayed above the head of 702). The first conversation message and the second conversation message may be displayed in different colors.

Figure 12 is an example diagram illustrating a situation in which a conversation message being displayed is scrolled in augmented reality according to an embodiment of the present disclosure.

Referring to FIG. 12, the processor 120 of the electronic device 101 sends a plurality of conversation messages based on the first avatar 602 and the first conversation window corresponding to the first counterpart 601 in the augmented reality space 630. It can be displayed. For example, since the number of conversation messages that can be displayed in the augmented reality space 630 is limited to less than a certain number, the processor 120 may implement a scroll icon 1210 to check previously received conversation messages. For example, the processor 120 may detect a user's scroll action (e.g., a scroll gesture input moving up and down) corresponding to a plurality of conversation messages, and the previously received conversation may be detected as the scroll icon 1210 moves. A message can be displayed.

Figure 13 is an example diagram recommending that the latest message and group message be converted to an avatar in a chat window according to an embodiment of the present disclosure.

Referring to FIG. 13, when displaying a chat window 1310 (e.g., a chat message window) based on a message application (e.g., the message application 312 of FIG. 3), the processor 120 of the electronic device 101 uses an avatar. Dialog windows that can be converted to can be displayed with a highlight effect reflected. For example, when the processor 120 receives the latest message 1311 from the first user 1321, the processor 120 may display the latest message 1311 including a confirmation icon 1311-1 indicating whether the user has confirmed it. You can. The processor 120 may display conversation windows that can be converted to avatars in various colors in the augmented reality space in relation to each conversation message. For example, when avatars 1321, 1322, and 1323 representing users are set and can be displayed as avatars in the augmented reality space, the processor 120 may respond to each conversation message 1311, 1312, and 1313, It can be displayed in different colors.

According to one embodiment, the electronic device 101 uses an augmented reality display module (e.g., the display module 160 of FIG. 3) based on an augmented reality-related application (e.g., the augmented reality-related application 311 of FIG. 3). It can be displayed through When displaying a conversation window in the augmented reality space, the processor 120 of the electronic device 101 may distinguish conversation messages that can be implemented as an avatar and conversation messages that cannot be implemented as an avatar by color. The processor 120 can reflect the highlight effect in a conversation message that can be implemented as an avatar and display it in a form that recommends avatar conversion to the user.

According to one embodiment, the electronic device 101 can select a plurality of conversation messages from among the conversation messages included in the conversation window 1310 and simultaneously create a plurality of avatars corresponding to the plurality of conversation messages. For example, the processor 120 may display a notification message (e.g., a notification message for selecting a conversation message to convert to an avatar from a plurality of conversation messages) in connection with the creation of an avatar, and respond to the user's selection. Thus, an avatar can be created and displayed. The avatar's position may be determined randomly within the augmented reality space, or its position may be determined according to a set priority. For example, when displaying a plurality of avatars simultaneously, the processor 120 may display them separately for each group window or for each individual.

According to one embodiment, the processor 120 may set the posture and behavior of the avatar in various ways depending on the location where the avatar is placed. For example, when the avatar is placed on a sofa, the processor 120 may display the avatar as sitting on the sofa. For example, when an avatar is placed in a gym, the processor 120 may display the avatar wearing exercise clothes and performing a set exercise motion.

Figure 14 is an example of displaying in color a group chat window in which an avatar displayed in augmented reality according to an embodiment of the present disclosure is participating.

Referring to FIG. 14, a first conversation message 604 corresponding to the first avatar 602 and a second conversation message 704 corresponding to the second avatar 702 are displayed together in the augmented reality space 630. You can. For example, in the address book, the first avatar 602 may be stored in the “Family” category, and the second avatar 702 may be stored in the “Friends” category. The processor 120 may set the “Family” category as a first color and the “Friends” category as a second color. Referring to FIG. 14, the processor 120 displays the first area 603 where the first avatar 602 is located with a first color, and displays the second area 703 where the second avatar 702 is located with a second color. It can be displayed in color.

According to one embodiment, the processor 120 may display the categories to which the first avatar 602 and the second avatar 702 belong by color in the augmented reality space 630. According to one embodiment, the processor 120 displays category information to which each avatar belongs in different colors, thereby improving convenience when a user creates a group chat window. According to one embodiment, even if the color of the first area 603 and the color of the second area 703 are different, a new group chat window in which the first avatar 602 and the second avatar 702 participate can be created. .

Figure 15 is an example of determining the size of an avatar differently based on the size of the space in which the avatar can move according to an embodiment of the present disclosure.

Referring to FIG. 15, the processor 120 of the electronic device 101 may create various avatars of various sizes when implementing an avatar in an augmented reality space. For example, the size of the avatar may be a size set as a default value, or may include a size arbitrarily set by the user. According to one embodiment, the size of the avatar may be determined based on information related to the other person (e.g., gender, age, height, weight, physical characteristics, image information).

According to one embodiment, when the electronic device 101 implements an avatar in an augmented reality space, the processor 120 generates an area (1511, 1521, 1531) of the horizontal surface where the avatar can be located and a three-dimensional space based on the area ( 1512, 1522, 1532) can be confirmed. Referring to FIG. 15 , the processor 120 may check the first area 1511 of the horizontal plane where the first avatar 1501 can be located and the first space 1512 based on the first area 1511. The processor 120 may identify a second area 1521 of the horizontal plane in which the second avatar 1502 can be positioned and a second space 1522 based on the second area 1521, and the third avatar 1503 can be positioned. A third area 1531 of a possible horizontal plane and a third space 1532 based on the third area 1531 can be confirmed.

According to one embodiment, while a chat window is being displayed in an augmented reality space, the processor 120 may check the size of an avatar corresponding to a chat partner participating in the chat window in response to a user input to the chat window. The processor 120 may check the area of the horizontal plane where the avatar can be located and the three-dimensional space based on the area based on the confirmed size of the avatar in the augmented reality space. According to another embodiment, the processor 120 may determine the size of the avatar to be created based on the area of the confirmed horizontal plane and a three-dimensional space based on the area.

FIG. 16A is an example of implementing the shape of an avatar differently based on the size of the space in which the avatar can move according to an embodiment of the present disclosure. FIG. 16B is an example diagram illustrating a situation in which the shape of the avatar changes based on the size of the space in which the avatar can move when the avatar moves according to an embodiment of the present disclosure. FIG. 16C is an example diagram illustrating a situation in which the shape of an avatar changes in response to a gesture input input to the avatar according to an embodiment of the present disclosure.

Referring to FIGS. 16A to 16C , the processor 120 of the electronic device 101 may generate various types of avatars when implementing an avatar in an augmented reality space. For example, the processor 120 may structurally classify the avatar's body to generate a full-body avatar, an upper-body avatar, and/or a face avatar. According to one embodiment, the size of the avatar may be determined based on information related to the other person (e.g., gender, age, height, weight, physical characteristics (e.g., race), and image information).

Referring to FIG. 16A, when the electronic device 101 implements an avatar in an augmented reality space, the processor 120 determines the areas 1611, 1612, 1613, and 1614 of the horizontal plane where the avatar can be located and three-dimensional data based on the areas. You can check the spaces (1621, 1622, 1623, 1624). For example, the processor 120 may identify a first area 1611 of the horizontal plane where the first form avatar 1601 can be located and a first space 1621 based on the first area 1611. For example, the first form avatar 1601 may include an avatar in the form of a full body, or an avatar in which the form is implemented from the head to the legs. The processor 120 may identify a second area 1612 of the horizontal plane where the second form avatar 1602 can be positioned and a second space 1622 based on the second area 1612. For example, the second form avatar 1602 may include an avatar in the form of an upper body, and may include an avatar in which the shape is implemented from the head to the upper body. The processor 120 may identify a third area 1613 of the horizontal plane where the third form avatar 1603 can be positioned and a third space 1623 based on the third area 1613. For example, the third form avatar 1603 may include an avatar whose shape is implemented from head to shoulders. The processor 120 may identify a fourth area 1614 of the horizontal plane where the fourth form avatar 1604 can be located and a fourth space 1624 based on the fourth area 1614. For example, the fourth form avatar 1604 may include an avatar with head and face shapes implemented.

According to one embodiment, when implementing an avatar, the processor 120 may check the space in which the avatar will be located and determine the form of the avatar that can be placed in the confirmed space. The processor 120 may select one avatar from various types of avatars based on the confirmed space. For example, the processor 120 may preferentially select the avatar that occupies the most space among avatars that can be placed in the confirmed space.

Referring to FIG. 16b, while the third form avatar 1603 is being displayed in the augmented reality space, the processor 120 may move the third form avatar 1603 to another location in response to the user's gesture input. there is. For example, the gesture input is a first input for gripping the third form avatar 1603 implemented in the augmented reality space, and dragging the third form avatar 1603 in the gripped state to another location (e.g., destination point). A second input, and a third input releasing the grip of the third form avatar 1603 at another location. The processor 120 can confirm the space corresponding to the destination point and determine the shape of the avatar based on the confirmed space. For example, if the space corresponding to the destination point satisfies the first space 1621 based on the first area 1611 of the first form avatar 1601, the processor 120 determines the third form avatar 1603. can be displayed by changing it to the first form avatar 1601. For another example, if the space corresponding to the destination point satisfies the fourth space 1624 based on the fourth area 1614 of the fourth form avatar 1604, the processor 120 determines the third form avatar 1603 ) can be changed and displayed as a fourth form avatar 1604. The processor 120 may preferentially select the type of avatar that occupies the most space within the space corresponding to the destination point. During the second input of dragging the third shape avatar 1603, the processor 120 displays the shape of the avatar to be implemented based on the destination point as a preview image (e.g., a blurred image of the avatar, an afterimage of the avatar). It can also be displayed as one image). According to one embodiment, if the area of the destination point is smaller than the fourth area 1614 of the fourth form avatar 1604 while the second input is maintained, the processor 120 selects the third point of the third form avatar 1603. The color of the third area 1613 and the third space 1623 based on the third area 1613 can be changed. For example, the processor 120 may change the color of the third area 1613 or the third space 1623 to red to indicate that the third form avatar 1603 is unable to move to the destination point. there is.

Referring to FIG. 16C, while the third form avatar 1603 is being displayed in the augmented reality space, the processor 120 changes the third form avatar 1603 into another form of avatar (e.g. : Can be changed to the fourth form avatar (1604). For example, a gesture input may be a first input gripping with both hands at least two points 1651, 1652 (e.g., area, coordinates) associated with the third form avatar 1603, with the first input being held with both hands. In this case, both hands are dragged in an outward direction to increase the size of the third form avatar 1603, or both hands are dragged in an inward direction of the third form avatar 1603 to decrease in size. It may include a second input to move, and a third input to release the grip (eg, first input) on the at least two points 1651 and 1652. For example, the at least two points 1651 and 1652 may include an area and coordinates included in the third space 1623 related to the third form avatar 1603. For example, if the shape of the avatar can be changed while the second input is being performed, the processor 120 displays the changeable avatar as a preview image (e.g., an image displaying the avatar blurred, an image displaying an afterimage of the avatar). It can also be displayed as an image). If a third input (eg, an input that cancels the first input) is confirmed while the preview image is being displayed, the processor 120 may change the shape of the avatar based on the preview image. According to one embodiment, the processor 120 may change the third form avatar 1603 to the fourth form avatar 1604 in response to a gesture input.

According to one embodiment, at least two points 1651 and 1652 may be set in advance, and when a set gesture input (e.g., a gesture input for gripping at least two points) is confirmed, a three-dimensional shape that appears visually is displayed. It may be a user interface. For example, at least two points 1651 and 1652 may disappear from being visually visible after a set time has elapsed.

FIG. 17A is an example diagram illustrating a situation in which a first chat window is displayed in augmented reality based on a portable electronic device according to an embodiment of the present disclosure. FIG. 17B is an example diagram illustrating a situation in which a first avatar corresponding to a first conversation partner participating in a first conversation window is displayed in augmented reality based on a portable electronic device according to an embodiment of the present disclosure.

The portable electronic device 1701 of FIGS. 17A and 17B may be at least partially similar to the electronic device 101 of FIG. 1 or may further include other embodiments of the electronic device 101.

According to one embodiment, an electronic device that implements augmented reality may include a portable electronic device 1701. Referring to FIG. 17A, the portable electronic device 1701 can implement an augmented reality space 1730 based on a real-life environment and display it through the display module 1760 (e.g., the display module 160 in FIG. 3). can do. The processor of the portable electronic device 1701 (e.g., the processor 120 in FIG. 3) operates a message application (e.g., the processor 120 in FIG. 3) for displaying the first chat window 1711 while the augmented reality space 1730 is being displayed. A message application (312) can be executed. For example, the first chat window 1711 may be a chat window in which the first other party 1710 is participating.

Referring to FIG. 17B, the processor 120 creates a first avatar 1712 corresponding to the first counterpart 1710 in response to a user input (e.g., drag input, touch input) to the first chat window 1711. can be created, and the created first avatar 1712 can be displayed in the augmented reality space 1730.

According to one embodiment, electronic devices that implement the augmented reality world are not limited to wearable electronic devices (e.g., HMD devices, VST devices) and may include various types of electronic devices.

In a method of displaying an avatar according to an embodiment, a first chat window 620 with the first other party 601 may be displayed within the virtual space 630. The method according to one embodiment includes, in relation to the first chat window 620, a first avatar 602 corresponding to the first other party 601 in response to a first user input moving from a first point to a second point. You can check the size. The method according to one embodiment is based on the size of the confirmed first avatar 602, the area 603 of the horizontal plane where the first avatar 602 can be located, and the first avatar 602 in a three-dimensional space 660 based on the area. It is possible to check whether the avatar 602 can be placed. The method according to one embodiment is such that when the second point according to the first user input is included in the area 603 of the identified horizontal plane and the three-dimensional space 660 based on the area, the arrangement of the first avatar 602 is In response to availability, the first chat window 620 may be converted to the first avatar 602. The method according to one embodiment may display the first avatar 602 based on the identified horizontal plane area 603 and a three-dimensional space 660 based on the area.

The operation of converting the first chat window 620 into the first avatar 602 according to one embodiment is based on the area 603 of the confirmed horizontal plane and the first user input in the three-dimensional space 660 based on the area. When two points are located, an operation to check the shape of the first avatar 602 corresponding to the area 603 of the horizontal plane and the three-dimensional space 660, and based on the confirmed shape of the first avatar 602 An operation of converting the first chat window 620 into the first avatar 602 may be included.

The method according to one embodiment includes an operation of displaying a second chat window 720 with a second counterpart 701 while the first avatar 602 is being displayed in the virtual space 630, and a second chat window ( In relation to 720), in response to a second user input moving from a third point to a fourth point, confirming the size of the second avatar 702 corresponding to the second counterpart 701, the confirmed second Based on the size of the avatar 702, the operation of checking the area 703 of the horizontal plane where the second avatar 702 can be located and the three-dimensional space based on the area, and the fourth point according to the second user input is confirmed When included in the area 703 of the horizontal plane and a three-dimensional space based on the area, the operation of converting the second chat window 720 into the second avatar 702, and the confirmed area 703 of the horizontal plane and the area An operation of displaying the second avatar 702 based on the three-dimensional space may further be included.

The method according to one embodiment includes the operation of checking the distance between the first avatar 602 and the second avatar 702 within the virtual space 630, and if the checked distance is less than or equal to a set threshold, the first avatar ( 602) and an operation of creating a group chat window in which the second avatar 702 participates may be further included.

The method according to one embodiment includes the operation of integrating the first area where the first avatar 602 is located and the second area where the second avatar 702 is located into one when creating a group chat window, and within the virtual space 630. An operation of displaying at least one chat message included in the group chat window may be further included. According to one embodiment, the first color of the first area is determined based on the address book category containing the first counterpart 601, and the second area is determined by the first color based on the address book category containing the second counterpart 701. Two colors are determined, and the first area and the third area integrated based on the second area may be displayed in one color.

According to one embodiment, the area 603 of the horizontal surface where the first avatar 602 can be positioned is determined based on the horizontal and vertical lengths calculated corresponding to the size of the first avatar 602, and 3 based on the area The dimensional space 660 may be determined based on the determined horizontal length, determined vertical length, and height of the first avatar 602.

A method according to an embodiment includes an operation of displaying a group chat window in which a first party 601 and a second party 701 are participating within a virtual space 630, and, in relation to the group chat window, from a fifth point to a sixth point. In response to a third user input moving to, confirming the size of a group in which the first avatar 602 and the second avatar 702 are combined, based on the size of the confirmed group, the first avatar 602 and an operation of confirming the area of the horizontal plane where the second avatar 702 can be located and a three-dimensional space based on the area, and the sixth point according to the third user input is confirmed in the area of the horizontal plane and the three-dimensional space based on the area. If included, the operation of converting the group chat window into the first avatar 602 and the second avatar 702, and the first avatar 602 and the second avatar based on the area of the confirmed horizontal plane and a three-dimensional space based on the area. 2 An operation to display the avatar 702 may be further included.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play Store ^TM ) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

101: electronic device 120: processor
130: memory 160: display module
190: Communication module 311: Augmented reality-related application
312: Message application

Claims (20)

In the electronic device 101,
Communication module 190;
display module 160;
memory (130); and
Comprising a processor 120 operatively connected to the communication module 190, the display module 160, and the memory 130,
The processor 120,
Through the display module 160, a first conversation window 620 with the first other party 601 is displayed in the virtual space 630,
In relation to the first chat window 620, in response to a first user input moving from a first point to a second point, check the size of the first avatar 602 corresponding to the first other party 601, and ,
Based on the confirmed size of the first avatar 602, the area 603 of the horizontal plane where the first avatar 602 can be located and the first avatar 602 in the three-dimensional space 660 based on the area. Check whether placement of is possible,
If the second point according to the first user input is included in the confirmed area 603 of the horizontal plane and the three-dimensional space 660 based on the area, in response to the possible placement of the first avatar 602 Converting the first chat window 620 into the first avatar 602,
An electronic device that displays the first avatar (602) based on the identified horizontal plane area (603) and a three-dimensional space (660) based on the area. According to claim 1,
The processor 120,
When the second point according to the first user input is located in the confirmed area 603 of the horizontal plane and the three-dimensional space 660 based on the area, the area 603 of the horizontal plane and the three-dimensional space 660 ) Confirm the shape of the first avatar 602 corresponding to
An electronic device that converts the first chat window (620) into the first avatar (602) and the first chat message (604) based on the confirmed form of the first avatar (602). According to claims 1 and 2,
The processor 120,
While the first avatar 602 is being displayed in the virtual space 630, a second chat window 720 with a second counterpart 701 is displayed,
In relation to the second chat window 720, in response to a second user input moving from a third point to a fourth point, check the size of the second avatar 702 corresponding to the second counterpart 701, and ,
Based on the confirmed size of the second avatar 702, confirm the area 703 of the horizontal plane where the second avatar 702 can be located and a three-dimensional space based on the area,
If the fourth point according to the second user input is included in the confirmed area 703 of the horizontal plane and a three-dimensional space based on the area, converting the second chat window 720 to the second avatar 702 do,
An electronic device that displays the second avatar (702) based on the confirmed horizontal plane area (703) and a three-dimensional space based on the area. According to claims 1 to 3,
The processor 120,
Check the distance between the first avatar 602 and the second avatar 702 within the virtual space 630,
An electronic device that creates a group chat window in which the first avatar (602) and the second avatar (702) participate when the confirmed distance is less than or equal to a set threshold. According to claims 1 to 4,
The processor 120,
When creating the group chat window, the first area where the first avatar 602 is located and the second area where the second avatar 702 is located are integrated into one,
An electronic device that displays at least one conversation message included in the group conversation window within the virtual space (630). According to claims 1 to 5,
The first color of the first area is determined based on the address book category in which the first counterpart 601 is included,
The second color of the second area is determined based on the address book category in which the second counterpart 701 is included,
An electronic device in which a third area integrated based on the first area and the second area is displayed in one color. According to claims 1 to 6,
The processor 120,
Confirming user input for separating the first avatar 602 and the second avatar 702 integrated into the third area,
In response to the confirmed user input, dividing the third area into the first area of the first avatar (602) and the second area of the second avatar (702),
An electronic device that individually displays a first chat message included in the first chat window of the first avatar (602) and a second chat message included in the second chat window of the second avatar (702). According to claim 1,
The area 603 of the horizontal surface where the first avatar 602 can be positioned is determined based on the horizontal and vertical lengths calculated corresponding to the size of the first avatar 602,
The three-dimensional space 660 based on the area is determined based on the determined horizontal length, the determined vertical length, and the height of the first avatar 602. According to claim 1,
The processor 120,
If the position of the first chat window 620 moving by the user input is maintained for a set time in the confirmed horizontal plane area 603 and the three-dimensional space 660 based on the area, the first avatar ( Converting the first chat window 620 to the first avatar 602 in response to the arrangement of 602 being possible,
If the position of the first chat window 620 moving by the user input is not maintained for the time set in the confirmed area of the horizontal plane and the three-dimensional space based on the area, maintaining the first chat window 620 Electronic devices. According to claims 1 to 3,
The processor 120,
When the first avatar 602 and the second avatar 702 are displayed at least partially overlapping within the virtual space 630, the first chat window 620 and the second chat window 720 have priority. Check the rankings,
An electronic device that displays a highlight effect on an avatar corresponding to a chat window with relatively high priority. According to claim 1,
The processor 120,
When the first other party (601) receives a conversation message containing emotional information, an electronic device that displays an animation effect so that the first avatar (602) displays the emotional information based on the emotional information. According to claims 1 to 3,
The processor 120,
Displaying a group chat window in which the first counterpart 601 and the second counterpart 701 are participating within the virtual space 630,
In relation to the group chat window, in response to a third user input moving from a fifth point to a sixth point, confirm the size of the group in which the first avatar 602 and the second avatar 702 are combined,
Based on the confirmed size of the group, confirm the area of the horizontal plane where the first avatar 602 and the second avatar 702 can be located and a three-dimensional space based on the area,
If the sixth point according to the third user input is included in the confirmed area of the horizontal plane and a three-dimensional space based on the area, the group chat window is divided into the first avatar 602 and the second avatar 702. convert,
An electronic device that displays the first avatar (602) and the second avatar (702) based on the confirmed area of the horizontal plane and a three-dimensional space based on the area. According to claims 1 to 3,
The processor 120,
Determining the size and appearance of the first avatar 602 based on information related to the first other party 601,
An electronic device that determines the size and appearance of the second avatar (702) based on information related to the second counterpart (701). In a method of displaying an avatar,
An operation of displaying a first chat window 620 with a first counterpart 601 within a virtual space 630;
In relation to the first chat window 620, in response to a first user input moving from a first point to a second point, the size of the first avatar 602 corresponding to the first other party 601 is confirmed. movement;
Based on the confirmed size of the first avatar 602, the area 603 of the horizontal plane where the first avatar 602 can be located and the first avatar 602 in the three-dimensional space 660 based on the area. An operation to check whether placement of is possible;
If the second point according to the first user input is included in the confirmed area 603 of the horizontal plane and the three-dimensional space 660 based on the area, in response to the possible placement of the first avatar 602 An operation of converting the first chat window 620 into the first avatar 602; and
Displaying the first avatar 602 based on the confirmed horizontal plane area 603 and a three-dimensional space 660 based on the area; How to include . According to claim 14,
The operation of converting the first chat window 620 into the first avatar 602 is:
When the second point according to the first user input is located in the confirmed area 603 of the horizontal plane and the three-dimensional space 660 based on the area, the area 603 of the horizontal plane and the three-dimensional space 660 An operation of confirming the shape of the first avatar 602 corresponding to ); and
An operation of converting the first chat window 620 into the first avatar 602 based on the confirmed shape of the first avatar 602; How to include . The method of claims 14 to 15,
An operation of displaying a second chat window 720 with a second counterpart 701 while the first avatar 602 is being displayed within the virtual space 630;
In relation to the second chat window 720, in response to a second user input moving from a third point to a fourth point, the size of the second avatar 702 corresponding to the second counterpart 701 is confirmed. movement;
Based on the confirmed size of the second avatar 702, confirming an area 703 of a horizontal plane where the second avatar 702 can be located and a three-dimensional space based on the area;
If the fourth point according to the second user input is included in the confirmed area 703 of the horizontal plane and the three-dimensional space based on the area, the second chat window 720 is converted to the second avatar 702. The action of converting; and
Displaying the second avatar 702 based on the confirmed horizontal plane area 703 and a three-dimensional space based on the area; How to include more. The method of claims 14 to 16,
Checking the distance between the first avatar 602 and the second avatar 702 within the virtual space 630; and
If the confirmed distance is less than or equal to a set threshold, creating a group chat window in which the first avatar 602 and the second avatar 702 participate; How to include more. The method of claims 14 to 17,
When creating the group chat window, integrating a first area where the first avatar 602 is located and a second area where the second avatar 702 is located into one; and
An operation of displaying at least one conversation message included in the group conversation window within the virtual space 630; It further includes,
The first color of the first area is determined based on the address book category containing the first counterpart 601, and the second area is determined by a second color based on the address book category containing the second counterpart 701. is determined, and the third area integrated based on the first area and the second area is displayed in one color. According to claim 14,
The area 603 of the horizontal surface where the first avatar 602 can be positioned is determined based on the horizontal and vertical lengths calculated corresponding to the size of the first avatar 602,
A method in which the three-dimensional space 660 based on the area is determined based on the determined horizontal length, the determined vertical length, and the height of the first avatar 602. The method of claims 14 to 16,
Displaying a group chat window in which the first counterpart 601 and the second counterpart 701 are participating within the virtual space 630;
An operation of checking the size of a group in which the first avatar 602 and the second avatar 702 are combined in response to a third user input moving from a fifth point to a sixth point in relation to the group chat window. ;
Based on the confirmed size of the group, confirming an area of a horizontal plane where the first avatar 602 and the second avatar 702 can be located and a three-dimensional space based on the area;
If the sixth point according to the third user input is included in the confirmed area of the horizontal plane and a three-dimensional space based on the area, the group chat window is divided into the first avatar 602 and the second avatar 702. The action of converting; and
Displaying the first avatar 602 and the second avatar 702 based on the confirmed area of the horizontal plane and a three-dimensional space based on the area; How to include more.

Images