KR20240025431A - Electronic device and method for providing a conversation funtion using an avatar - Google Patents

Abstract

Embodiments of the present disclosure relate to an electronic device and method that provides a conversation function using an avatar in a virtual space. For this purpose, the electronic device may output a first conversation screen using an avatar based on data collected from an external electronic device based on a network environment. The electronic device may identify a speaker among the attendees based on the collected data, and configure a second conversation screen by switching the camera angle based on the speaker avatar corresponding to the identified speaker.

Description

Electronic device and method for providing a conversation function using an avatar {ELECTRONIC DEVICE AND METHOD FOR PROVIDING A CONVERSATION FUNTION USING AN AVATAR}

Embodiments of the present disclosure relate to an electronic device and method that provides a conversation function using an avatar in a virtual space.

Electronic devices such as smart phones, mobile phones, and tablet devices can be representative examples of electronic devices that are easy for users to carry. Application programs to provide various functions may be installed on the electronic device. The electronic device may execute an installed application program in response to a user's manipulation. The electronic device may display information according to the execution of an application program through a display.

The electronic device allows users to use functions such as games, shopping, or chatting using avatars in a virtual space based on a network environment. The avatar function provided by the electronic device in a virtual space based on a network environment is limited to the avatar moving on behalf of the user or having a conversation using voice or text.

Therefore, there may be a lack of providing a natural communication experience to the user by reflecting the user's characteristics, such as movement or gestures during conversation, in the avatar.

An embodiment of the present disclosure provides an electronic device and a control method for adjusting the angle or view angle of a camera for providing a conversation screen based on an avatar corresponding to a speaker in a virtual space based on a network environment. can do.

According to one embodiment, an electronic device includes a front camera, a camera module configured to output an electrical signal by photographing a subject, and a display panel or touch panel, and displays an image on the display panel, or A display module configured to output an electrical signal according to a touch of the touch panel, a communication module configured to communicate with an external electronic device based on a network environment, and an electrical connection between the camera module, the display module, or the communication module. It may include at least one processor configured to control the operation of the camera module, the display module, or the communication module.

According to one embodiment, the at least one processor displays the avatar of at least one participant among a plurality of participants including the user of the electronic device in a virtual space provided for a conversation function, and displays the avatar of at least one participant among the plurality of participants. A speech indicator indicating the speech state is displayed in response to an utterance by a speaker, and the display module can be controlled to change and display the gaze of the avatar in the virtual space based on the utterance by the at least one speaker.

According to one embodiment, a method of providing a conversation function using an avatar in an electronic device includes displaying an avatar of at least one participant among a plurality of participants including a user of the electronic device in a virtual space provided for the conversation function. and, displaying an utterance indicator indicating an utterance state in response to an utterance by at least one speaker among the plurality of participants, and changing the gaze of the avatar in the virtual space based on the utterance by the at least one speaker. It may include a display action.

According to one embodiment, an electronic device includes a front camera and may include a camera module configured to output an electrical signal by photographing a subject. The electronic device may include a display panel or a touch panel, and may include a display module configured to display an image on the display panel or output an electrical signal according to a touch of the touch panel. The electronic device may include a communication module configured to communicate with an external electronic device based on a network environment. The electronic device may include at least one processor configured to control the operation of the camera module, the display module, or the communication module by electrical connection with the camera module, the display module, or the communication module.

According to one embodiment, the at least one processor may identify that a speaker has changed among attendees who participated in a conversation using an avatar based on data collected from the external electronic device through the communication module. The at least one processor may configure a first conversation screen with a changed camera angle based on the speaker avatar corresponding to the changed speaker. The at least one processor may control the display module to display the first conversation screen on the display panel.

According to one embodiment, a method of providing a conversation function using an avatar in an electronic device may include outputting a first conversation screen using an avatar based on data collected from an external electronic device based on a network environment. there is. The method may include identifying a speaker among attendees based on the collected data. The method may include configuring a second conversation screen by switching the camera angle based on the speaker avatar corresponding to the identified speaker. The method may include converting the first conversation screen to the second conversation screen and outputting the second conversation screen.

According to an embodiment of the present disclosure, an electronic device can provide a conversation screen centered on an avatar corresponding to a speaker in a virtual space, allowing a user to obtain a natural experience as if meeting the other person in person and having a conversation.

The technical problems to be achieved in the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not mentioned above may be derived from the exemplary embodiments of the present disclosure by those skilled in the art. there is.

Effects that can be obtained from the exemplary embodiments of the present disclosure can be clearly derived and understood by those skilled in the art from the following description. That is, unintended effects resulting from implementing the exemplary embodiments of the present disclosure may also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

Figure 1 is a configuration diagram of a system for providing an avatar conversation function in a network environment, according to an embodiment.
Figure 2 is a control flow diagram for providing an avatar conversation function in a system, according to one embodiment.
Figure 3 is a signal flow diagram for providing an avatar conversation function in a system, according to one embodiment.
Figure 4 is a control flowchart to be performed for an avatar-based conversation function in an electronic device, according to one embodiment.
Figure 5 is a control flowchart to be performed for an avatar-based conversation function in an operation server, according to one embodiment.
Figure 6 is a signal flow diagram for performing an avatar conversation subroutine in a system, according to one embodiment.
Figure 7 is a signal flow diagram for performing an avatar conversation subroutine in a system, according to one embodiment.
Figure 8 is a signal flow diagram for performing an avatar conversation subroutine in a system, according to one embodiment.
Figure 9 is a signal flow diagram for performing an avatar conversation subroutine in a system, according to one embodiment.
Figure 10 is a control flowchart for configuring a conversation screen for an avatar conversation function in an electronic device, according to an embodiment.
11 is a block diagram of an electronic device, according to an embodiment.
Figure 12 is an example diagram of a user interface (UI) for each procedure for opening an avatar-based chat room in an electronic device, according to an embodiment.
FIG. 13 illustrates an example of a user interface provided before entering a chat room for an avatar-based chat function in an electronic device, according to an embodiment.
Figure 14 illustrates an example of angle switching in response to a speaker change in an electronic device, according to an embodiment.
FIGS. 15A and 15B illustrate an example of angle switching corresponding to manual manipulation in an electronic device, according to an embodiment.
Figure 16 illustrates an example of switching the angle of view in response to setting requirements in an electronic device, according to an embodiment.
FIG. 17 illustrates an example of view (angle of view) switching in response to manual manipulation in an electronic device, according to an embodiment.
FIGS. 18A and 18B illustrate an example of mirroring a user's movements on an avatar using an image captured by a camera in an electronic device, according to an embodiment.
FIGS. 19A, 19B, and 19C show examples of movements that are preset in an electronic device and can be applied to an avatar, according to an embodiment.
Figures 20A to 20E illustrate an example of changing the screen shape in consideration of the number of participants in an electronic device, according to an embodiment.
FIG. 21 illustrates an example of calling an opponent's avatar through manual manipulation in an electronic device, according to an embodiment.
FIG. 22 illustrates an example of recording a conversation screen at the request of a user in an electronic device, according to an embodiment.
Figure 23 is a signal flow diagram for providing a gift event during a conversation using an avatar in a system, according to an embodiment.
Figures 24A to 24I are exemplary diagrams of a user interface (UI) for each procedure for delivering a gift on a conversation screen using an avatar, according to an embodiment.
Figure 25 is a block diagram of an electronic device in a network environment, according to various embodiments.
In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components.

Hereinafter, with reference to the drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily practice them. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components. Additionally, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

Figure 1 is a configuration diagram of a system 100 for providing an avatar conversation function in a network environment, according to one embodiment.

Referring to FIG. 1, a system 100 for providing an avatar conversation function may include a plurality of electronic devices (eg, electronic devices 2501, 2502, and 2504 of FIG. 23). The plurality of electronic devices 2501, 2502, and 2504 may include a plurality of electronic devices 110 or a plurality of wearable devices 120. The system 100 may include one or multiple servers 130. The plurality of electronic devices 110 may transmit or receive data to and from the server 130 through a network 140 that provides a communication channel using wireless or wired resources.

Referring to FIG. 1, in one embodiment, the system 100 for providing an avatar conversation function may include a plurality of electronic devices (e.g., electronic devices 2501, 2502, and 2504 of FIG. 23) or a server 130. You can. The plurality of electronic devices 2501, 2502, and 2504 may include a plurality of electronic devices 110 or a plurality of wearable devices 120. The plurality of electronic devices 110 may include first to nth electronic devices 111, 113, and 115. The first to nth electronic devices 111, 113, and 115 may be the same or different types of devices. The first to nth electronic devices 111, 113, and 115 can install an application program, such as a smartphone or a tablet PC, run the installed application program, and use the functions provided by the executed application program. It can be a device that can The plurality of wearable devices 120 may include various types of wearable devices such as a smart watch 121, wireless earphones 123, or smart glasses 125. The plurality of wearable devices 120 may manage information such as the user's physical condition, amount of exercise, or movement, or provide the user with functions such as listening to music or virtual space. The server 130 may include heterogeneous servers 131 and 133 that provide the same or different functions or functions. The heterogeneous servers 131 and 133 included in the server 130 may include an authorization server, a matching server, a present server, or an asset server. . The authentication server may confirm the user's access qualifications and issue access rights. The matching server may provide an API that allows users to create and register an operating server. The matching server may provide a list so that users can participate in the operation server. The matching server can manage status information of users. The matching server may send an invitation message to the user. The operating server can collect real-time data. Users connected to the operating server can upload or download data. The operating server may be implemented to run on the electronic device 110 or an external server. The asset server can provide data such as mesh and texture that can render an avatar or background. The heterogeneous servers may be modularized and added to one server.

For convenience of description below, the plurality of electronic devices 111, 113, and 115 will be collectively referred to as the electronic device 110, and the plurality of wearable devices 121, 123, and 125 will be collectively referred to as the wearable device 120. .

The electronic device 110 receives data for the avatar conversation function through a network 140 (e.g., the first network 2598 and the second network 2599 in FIG. 23) that operates based on one or more protocols. It can be exchanged with other electronic devices or servers 130. The one or more protocols, for example, enable communication in a network environment such as a short-range communication network (e.g., the first network 2598 in FIG. 23) or a long-distance communication network (e.g., the second network 2599 in FIG. 23). It may be a protocol prepared to perform.The protocol prepared for the short-range communication network may include, for example, Bluetooth, WiFi direct, or IrDA, etc. The long-distance communication network may include, for example, a legacy cellular network, 5G networks, next-generation communications networks, the Internet, or telecommunications networks such as computer networks (e.g., LAN or WAN).

The electronic device 110 can be connected to the wearable device 120 through a predetermined connection procedure. The connection procedure may be performed according to a protocol established for each communication method. The protocol provided for each communication method allows communication in a network environment such as, for example, a short-range communication network (e.g., the first network 2598 in FIG. 23) or a long-distance communication network (e.g., the second network 2599 in FIG. 23). It may be a protocol prepared to perform.The protocol prepared for the short-range communication network may include, for example, Bluetooth, WiFi direct, or IrDA, etc. The long-distance communication network may include, for example, a legacy cellular network, 5G networks, next-generation communications networks, the Internet, or telecommunications networks such as computer networks (e.g., LAN or WAN).

The electronic device 110 may collect data for the avatar conversation function from another electronic device, a wearable device 120, or a server 130, which is an external electronic device, based on the network environment 140. The electronic device 110 may output a first conversation screen using an avatar through a display based on the collected data. The electronic device 110 can identify speakers among attendees based on the collected data. The electronic device 110 may configure the second conversation screen by switching the camera angle, view angle, or screen view based on the speaker avatar corresponding to the identified speaker. The electronic device 110 may change the conversation screen for the avatar conversation function from the first conversation screen to the second conversation screen and output it.

The electronic device 110 may acquire the voice characteristics of the participants based on the participants' voice data included in the collected data. The electronic device 110 may predict a participant having an acquired voice characteristic using information about a preset reference voice characteristic. The electronic device 110 may determine the avatar of the predicted participant among the avatars participating in the conversation as the speaker avatar.

When a chat room using an avatar is opened and conversations are conducted by attendees, the electronic device 110 may collect the voice data of the attendees at a certain time or at a certain cycle. The electronic device 110 may use the collected voice data to obtain information about the standard voice characteristics of each participant. The electronic device 110 may register information regarding the acquired standard voice characteristics for each participant.

The electronic device 110 may configure a conversation screen with a switched camera angle based on an electrical signal provided by a user's swipe operation on the touch screen. The electronic device 110 may output a conversation screen through a display.

The electronic device 110 may configure a conversation screen with a switched camera angle based on an electrical sensing signal obtained by a user's left/right or up/down movement. The electronic device 110 may output a conversation screen through a display.

The electronic device 110 may adjust or change the angle of view of the conversation screen based on the collected data. The electronic device 110 may configure a conversation screen to which an adjusted or changed viewing angle is applied. The electronic device 110 may output a conversation screen through a display.

If the speech time by a specific avatar does not exceed the threshold time, the electronic device 110 may change the conversation screen to a conversation screen in which the specific avatar is zoomed in. The electronic device 110 may switch to a conversation screen that zooms in on a specific object in response to the user's swipe action. If the speech time by a specific avatar exceeds a threshold time, the electronic device 110 may switch to a conversation screen with the specific avatar zoomed out. The electronic device 110 may switch to a conversation screen in which a specific object is zoomed out in response to the user's swipe action.

The electronic device 110 acquires a facial expression or gesture from the user's image captured by the front camera, and mirrors the acquired facial expression or gesture to the user's avatar included in the conversation screen. You can. The electronic device 110 may obtain the facial expressions or gestures of the attendees from the collected data, and mirror the acquired facial expressions or gestures of the attendees on the avatars of the attendees included in the conversation screen. .

The electronic device 110 may adjust the angle of view in consideration of the number of attendees and output a conversation screen prepared by the adjusted angle of view. When the electronic device 110 detects that the user has touched the avatar of one of the attendees, it may transmit a signal to call an external electronic device corresponding to the touched avatar. The electronic device 110 may extract images corresponding to a conversation based on the collected data, create a conversation record image using the extracted images, and store the created conversation record image.

The server 130 may open a chat room to provide an avatar-based conversation function in response to a request from the electronic device 110. The server 130 may request entry into the chat room from the electronic device 110 of the guest to be invited to the opened chat room. The server 130 can configure a conversation screen suitable for each electronic device 110 based on data collected from the electronic device 110 and transmit information about the constructed conversation screen to the corresponding electronic device 110. there is.

Figure 2 is a control flow diagram for providing an avatar conversation function in a system (eg, system 100 of Figure 1), according to one embodiment.

Referring to FIG. 2, in one embodiment, the system 100 may perform a procedure for opening a chat room in operation 210. In one embodiment, the procedure for opening a chat room may include a chat room creation procedure to prepare a chat room or a chat room entry procedure for participants to access the chat room. The participants may include, for example, a host who opened a chat room or at least one guest who has been requested to participate in the chat room.

The chat room creation procedure is performed by an electronic device corresponding to a host (e.g., the first electronic device 111 in FIG. 1) (hereinafter referred to as “host terminal 111”) operating a server (e.g., the operation server 130 in FIG. 1). )) may include a request procedure for requesting the opening of a chat room. The chat room creation procedure may include a creation procedure in which the operation server 130 creates a chat room in response to the request. The chat room may, for example, correspond to a virtual space where avatars can converse on behalf of participants.

In the chat room entry procedure, the operation server 130 connects an electronic device (e.g., the second electronic device 113 or the nth electronic device 115 of FIG. 1) corresponding to the guest requested to participate in the conversation by the host terminal 111. ) (hereinafter referred to as “guest terminals 113, 115”) may include a call procedure. The chat room entry procedure may include a response procedure in which the guest terminals 113 and 115 enter the chat room in response to a call from the operation server 130. The host terminal 111 and the guest terminal 113 or 115 may be collectively referred to as 'participating terminals (eg, electronic devices 110 in FIG. 1)' that have entered the chat room for conversation.

In one embodiment, system 100 may perform an avatar-based conversation procedure at operation 220. The avatar-based conversation procedure may include an avatar setting procedure for setting a host avatar to replace the host or a guest avatar to replace the guest in the chat room. The host avatar or the guest avatar may be collectively referred to as a 'participation avatar' that replaces the participant who participated in the conversation. Information about the participating avatar may be shared between participating terminals 110, for example.

The avatar-based conversation procedure may include a screen configuration procedure for configuring a conversation screen on which a conversation will be held by participating avatars. For example, the conversation screen may be configured for each participant based on information collected by the operation server 130 from participants (eg, host or guest) through the participation terminal 110. A conversation screen configured for each participant by the operation server 130 may be provided to the participating terminal 110. A conversation screen configured for each participant may be prepared, for example, from the viewpoint of the participant. For example, the conversation screen may be configured based on information collected by a participating terminal (e.g., host terminal 111) from another participating terminal (e.g., guest terminal 113 or 115). The conversation screen constructed by the participation terminal 110 can reflect the user's request in real time, regardless of the opinions of other participants.

The avatar-based conversation procedure may include movements of a participating avatar on a conversation screen (e.g., facial expressions, gaze, or body movements such as hands, arms, neck, or legs) or a conversation performance procedure for performing a conversation between participating avatars. there is. Conversations between the participating avatars may be conducted using natural language, for example, the participants' voices or text converted from the participants' voices. In one embodiment, in the conversation performance procedure, at least one speaker among the participants may be identified, and the conversation screen may be changed based on the at least one speaker's avatar (hereinafter referred to as “speaker avatar”).

The change in the conversation screen may, for example, constitute a conversation screen in which the gaze of the avatar corresponding to the listener (hereinafter referred to as “listener avatar”) is changed to the direction of the speaker avatar. The conversation screen is changed by, for example, reflecting (e.g. mirroring) the movements of the participants (e.g. facial expressions, gaze or body movements such as hands, arms, neck or legs). It may constitute a . The participants' movements can be obtained, for example, from images of users captured through a camera. The participants' movements may be obtained, for example, by sensing information measured by a sensor (e.g., a gyroscope sensor) provided in the electronic device 110. The participants' movements may be obtained, for example, For example, it can be obtained through sensing information measured by a sensor (eg, a gyroscope sensor) provided in a wearable device (eg, wearable device 120 of FIG. 1).

For example, changing the conversation screen may mean configuring the conversation screen by changing the camera angle so that the speaker avatar is positioned at the center. For example, changing the conversation screen may mean configuring the conversation screen by rotating the camera angle in a specific direction (e.g., left/right, up/down, clockwise, or counterclockwise) around the speaker avatar. For example, the camera angle may correspond to a direction in which participants are seen having a conversation in a virtual space.

For example, changing the conversation screen may mean configuring the conversation screen by changing the angle of view. The angle of view may be, for example, a narrow angle of view in which the conversation screen focuses on a specific object (eg, a speaker avatar), or a wide angle of view in which the conversation screen entirely includes objects (eg, participating avatars). The operation of changing the angle of view for forming the conversation screen to a narrow angle of view may correspond to a zoom-in operation. The operation of changing the angle of view for configuring the conversation screen to a wider angle of view may correspond to a zoom-out operation.

In the above-described examples, the angle or angle of view that changes the conversation screen may be, for example, a condition that changes the view of the conversation screen. Conditions for changing the view of the conversation screen may include the screen shape, such as a horizontal screen or a vertical screen. The view type, which is the form of the screen, may be determined considering the way the user holds the electronic device and/or the number of participating avatars to be included in the screen.

In one embodiment, the system 100 may perform a procedure for closing the chat room in operation 230. In one embodiment, a procedure for closing a chat room may include an exit procedure in which a specific participant leaves or a closure procedure in which all participants leave. The participants may include, for example, a host who opened a chat room or at least one guest participating in the chat room.

In the exit procedure, for example, a guest terminal corresponding to a specific participant among participants participating in a conversation (e.g., the first electronic device 113 in FIG. 1) requests exit from the operation server 130, This can be performed by the operation server 130 approving this. Before approving the guest terminal 113's exit, the operation server 130 may ask other participating terminals (eg, the host terminal 111) whether they agree to the participant's exit.

The closing procedure is, for example, a host terminal (e.g., the first electronic device 1111 in FIG. 1) requests the operation server 130 to close the chat room, and the operation server 130 approves the request. It can be done. The operation server 130 may receive confirmation of the chat room closure from the guest terminal (e.g., the nth guest terminal 115) that remains without leaving before approving the chat room closure.

FIG. 3 is a signal flow diagram for providing an avatar conversation function in a system (eg, system 100 of FIG. 1), according to one embodiment.

Referring to FIG. 3 , in operation 311, the host terminal 111 may request the operation server 130 to open a chat room and/or invite a guest. For example, in order to request the opening of a chat room, the host terminal 111 may provide information about the chat room to be opened (e.g., chat room name or password) to the operation server 130. For example, in order to invite a guest to a chat room, the host terminal 111 may provide information about the guest to be invited (eg, participant phone number or participant ID) to the operation server 130.

Upon receiving a request for opening and inviting a chat room from the host terminal 111, the operation server 130, in operation 313 or 315, sends a first guest terminal 113 (e.g., FIG. A request to enter the chat room may be made through the second electronic device 113 of FIG. 1) and/or the n-th guest terminal 115 (e.g., the n-th electronic device 115 of FIG. 1). For example, the operation server 130 may include information about the chat room (e.g., chat room name) in a message requesting entry to be transmitted to the first guest terminal 113 and/or the n-th guest terminal 115. Can be included. The information about the chat room may be link information (eg, URL information) through which one can enter the chat room.

The first guest terminal 113 and/or the nth guest terminal 115, which are guest terminals invited to the chat room, may respond to entry into the chat room opened by the operation server 130 in operation 317 or operation 319. there is. For example, the first guest terminal 113 and/or the n-th guest terminal 115 may display link information (e.g., URL information) provided by the operation server 130 for an admission request. When the user touches the link information output through the display, a response message agreeing to entry can be transmitted to the operation server 130.

In operation 321, the operation server 130 completes responses from all invited guest terminals (e.g., the first guest terminal 113 and/or the n-th guest terminal 115) after opening the chat room. , you can open a chat room and respond to an invitation using the host terminal 111. For example, when a conversation is possible, the operation server 130 may transmit information about the chat room and/or information about attending guests to the host terminal 111.

Participating terminals 110 (e.g., host terminal 111, first guest terminal 113, and/or n-th guest terminal 115) entering the chat room perform conversation by avatar in operation 323. You can. In order to conduct a conversation using the avatar, the operation server 130 may set a participating avatar including a host avatar or a guest avatar. The operation server 130 may share information about the participating avatar with the participating terminals 110.

According to one embodiment, the participating terminals 110 (e.g., the host terminal 111, the first guest terminal 113, and/or the n-th guest terminal 115) are collected through the operation server 130, or The conversation screen can be configured based on information collected directly from other participating terminals. The conversation screen constructed by the participation terminal 110 can reflect the user's request in real time, regardless of the opinions of other participants. For example, the participating terminals 110 may change the conversation screen in consideration of events such as speaker change, participant's manual operation, or participant's movement. In order to change the conversation screen, the participating terminals 110 can exchange information about the event with each other. In one embodiment, the participating terminals 110 may identify the speaker currently speaking based on the participant's voice data or an image taken of the participant. The participating terminals 110 may configure a new conversation screen by changing the angle and/or view angle based on the identified speaker. In one embodiment, the participating terminals 110 may configure a new conversation screen by changing the angle and/or view angle in response to the participant's (or user's) manual manipulation. In one embodiment, the participating terminals 110 acquire the movements of participants (e.g., hosts or guests) from images taken by themselves or images taken by other participating terminals, and display a conversation screen reflecting the obtained movements. can be newly configured. In one embodiment, the participating terminals 110 acquire the movements of participants (e.g., host or guest) based on an internal sensing signal or an external sensing signal provided by another participating terminal, and conduct a conversation reflecting the obtained movements. The screen can be reconfigured.

According to one embodiment, the operation server 130 may configure a conversation screen for each participant based on information collected from the participating terminals 110. The operation server 130 may provide a conversation screen configured for each participant to the participating terminal 110. For example, the operation server 130 may change the conversation screen for each participant by considering events such as a change in speaker, manual manipulation of the participant, or movement of the participant. In order to change the conversation screen, the operation server 130 may collect information about events occurring in the participating terminals 110. In one embodiment, the operation server 130 may identify the speaker currently speaking based on the participant's voice data or an image taken of the participant. The operation server 130 may configure a new conversation screen for each participant by changing the angle and/or angle of view based on the identified speaker. In one embodiment, the operation server 130 may configure a new conversation screen to be provided to the participating terminals 110 by changing the angle and/or view angle in response to the manual operation of the participant (or user). In one embodiment, the operation server 130 acquires the movements of participants (e.g., host or guest) from images captured by the participating terminals 110, and refreshes the conversation screen for each participant reflecting the obtained movements. It can be configured. In one embodiment, the operation server 130 acquires the movements of participants (e.g., hosts or guests) based on sensing signals provided by the participating terminals 110, and creates a conversation for each participant reflecting the obtained movements. The screen can be reconfigured. The operation server 130 may provide information about the newly configured conversation screen to the participating terminals 110.

When a user requests to leave the chat room, the guest terminal (eg, the first guest terminal 113) may request the operation server 130 to leave in operation 325. When the operation server 130 receives an exit request from the first guest terminal 113, in operation 327, the operation server 130 sends an exit approval request to the host terminal ( 111). In operation 329, the host terminal 111 may transmit an exit approval response agreeing to the exit of the first guest terminal 113 to the operation server 130. Although not shown, the operation server 130 may transmit an exit approval request to inform the nth guest terminal 115, which is another guest terminal, that the first guest terminal 113 has requested to leave. The n-th guest terminal 115 may transmit an exit approval response agreeing to the exit of the first guest terminal 113 to the operation server 130.

When the user requests to end the conversation, the host terminal 111 may request the operation server 130 to close the chat room in operation 333. Upon receiving a request to close the chat room from the host terminal 111, the operation server 130 may notify the nth guest terminal 115, which is the remaining guest terminal, of the chat room closure in operation 335. In operation 337, the n guest terminal 115 may transmit a response confirming that the chat room is closed to the operation server 130. In operation 339, the operation server 130 may notify the host terminal 111 that the chat room is closed.

FIG. 4 is a control flowchart to be performed for an avatar-based conversation function in an electronic device (eg, the electronic device 110 of FIG. 1 ), according to an embodiment.

Referring to FIG. 4 , in operation 411, the electronic device 110 may determine whether an event requesting the opening of a chat room by the user occurs. The event requesting the opening of a chat room is, for example, an application that provides an avatar-based conversation function is executed, and a request is made to open a chat room through a user interface (UI: user interface) provided by the executed application. It can be generated by inputting information.

When a request is made to open a chat room by a user, the electronic device 110 sends a request for opening a chat room and inviting participants to an operation server (e.g., the server 130 in FIG. 1 or the operation server 130 in FIG. 3) in operation 413. It can be sent to . For example, in order to request the opening of a chat room, the electronic device 110 may provide information about the chat room to be opened (e.g., chat room name or password) to the operation server 130. For example, in order to invite a guest to a chat room, the electronic device 110 may provide information about the guest to be invited (eg, a participant's phone number or a participant's ID) to the operation server 130.

In operation 415, the electronic device 110 may determine whether opening a chat room using the avatar-based chat function was successful. Success in opening the chat room means, for example, that the operation server 130 opens the chat room and then invites all guest terminals (e.g., the first guest terminal 113 in FIG. 3 and/or the n-th guest terminal ( This is because the response by 115)) has been completed. When the electronic device 110 receives a notification from the operation server 130 that the chat room has been successfully opened, the electronic device 110 may receive information about the chat room and/or information about the guests present.

In operation 417, the electronic device 110 may determine whether a request to enter a chat room established to use the avatar-based conversation function is received from the operation server 130. The entry request may be provided from the operation server 130, for example, by inviting another electronic device to a chat room that another electronic device has requested to open.

The electronic device 110 may transmit an admission response corresponding to the admission request from the operation server 130 to the operation server 130 in operation 419. For example, when transmitting an entry response, the electronic device 110 may transmit information about the avatar to be used by the electronic device 110 to the operation server 130.

If the electronic device 110 succeeds in opening a chat room or responds to an entry request, in operation 421, a participating terminal (e.g., a guest terminal (e.g., the first guest terminal 113 in FIG. 3) is displayed on a chat screen provided in the virtual space. ) and/or an avatar-based conversation function with the nth guest terminal 115) may be provided to the user.

According to one embodiment, the electronic device 110 may configure a conversation screen based on information collected through the operation server 130 or directly collected from other participating terminals. The conversation screen constructed by the electronic device 110 may reflect the user's request in real time, regardless of the opinions of other participants. For example, the electronic device 110 may change the conversation screen in consideration of events such as a change in speaker, manual manipulation of a participant, or movement of a participant. In order to change the conversation screen, the electronic device 110 sends information about the event to participating terminals (e.g., guest terminals (e.g., the first guest terminal 113 and/or the n-th guest terminal 115 in FIG. 3 ). )) can be exchanged with each other. The electronic device 110 can identify who the speaker currently speaking is, for example, based on the participant's voice data or an image taken of the participant. The electronic device 110 may configure a new conversation screen by changing the angle and/or angle of view based on the identified speaker. For example, the electronic device 110 may respond to manual manipulation by a participant (or user). In response, the conversation screen can be newly configured by changing the angle and/or angle of view. For example, the electronic device 110 may select participants (e.g. : The movement of the host or guest) can be acquired, and a conversation screen reflecting the obtained movement can be newly configured. The electronic device 110 may use, for example, an internal sensing signal or an external sensing provided by another participating terminal. Based on the signal, the movements of the participants (e.g., host or guest) can be acquired, and a new conversation screen reflecting the obtained movements can be configured.

According to one embodiment, the electronic device 110 may output a conversation screen using an avatar-based conversation function based on information about the conversation screen provided from the operation server 130. The information about the conversation screen provided from the operation server 130 may be, for example, information about the conversation screen configured for each participant based on information collected by the operation server 130 from the participating terminals 110. there is. For example, the operation server 130 may change the conversation screen for each participant in consideration of events such as a change in speaker, manual manipulation of the participant, or movement of the participant. The electronic device 110 may collect information about events occurring internally and transmit it to the operation server 130 so that the operation server 130 can change the conversation screen. For example, the electronic device 110 may transmit the participant's voice data or a photographed image of the participant to the operation server 130. The transmitted voice data of the participant or an image taken of the participant may be used by the operation server 130 to identify who the speaker is.

The conversation screen output by the electronic device 110 based on information about the conversation screen provided by the operation server 130 may be a conversation screen whose angle and/or view angle is changed based on the speaker. For example, the conversation screen output by the electronic device 110 may be a conversation screen whose angle and/or view angle is changed in response to manual manipulation by a participant (or user). The conversation screen output by the electronic device 110 may be, for example, a conversation screen that reflects the movements of participants (e.g., host or guest) obtained from images captured by the participating terminals 110. . The conversation screen output by the electronic device 110 is, for example, a conversation screen reflecting the movements of participants (e.g., host or guest) obtained based on sensing signals provided by the participating terminals 110. It can be.

In operation 423, the electronic device 110 may determine whether a conversation termination request event or a chat room closure request event occurs by the user. For example, when the electronic device 110 is a guest terminal, the conversation end request event may occur for the purpose of leaving the chat room in which the user is currently participating. For example, if the electronic device 110 is a host terminal, the chat room closure request event may occur for the purpose of closing the chat room in which the user is currently participating.

The electronic device 110 may terminate the avatar-based conversation function when a conversation termination or chat room closure request event occurs. Terminating the avatar-based conversation function may, for example, correspond to terminating an application program that provides the avatar-based function.

FIG. 5 is a control flowchart to be performed for an avatar-based conversation function in an operation server (eg, server 130 of FIG. 1 ), according to an embodiment.

Referring to FIG. 5, in operation 511, the operation server 130 receives a request for opening a chat room and inviting participants from a host terminal (e.g., the first electronic device 111 in FIG. 1 or the host terminal 111 in FIG. 2). You can determine whether it is accepted. For example, the operation server 130 may receive information about the chat room to be opened (eg, chat room name or password) from the host terminal 111. For example, the operation server 130 may receive information about guests to be invited (eg, participant phone number or participant ID) from the host terminal 111.

Upon receiving the request for opening the chat room and inviting the participants, the operation server 130 may open the chat room in operation 513. For example, the operation server 130 may open a chat room using information about the chat room (eg, chat room name or password) received from the host terminal 111.

In operation 515, the operation server 130 enters a chat room with a guest terminal (e.g., the first guest terminal 113 or the nth guest terminal 115 in FIG. 2) requested to participate by the host terminal 111. You may request entry. For example, the operation server 130 may request participant entry using information about the guest (eg, participant phone number or participant ID) received from the host terminal 111. When requesting entry to the guest terminals 113 and 115, the operation server 130 may include information about the corresponding chat room (eg, chat room name). The information about the chat room may be link information (eg, URL information) through which one can enter the chat room.

In operation 517, the operation server 130 may determine whether all guest terminals 113 and 115 that have requested entry have completed entering the opened chat room. If only some guest terminals have entered, the operation server 130 may wait until the remaining guest terminals enter. The operation server 130 may repeatedly transmit an entry request up to a predetermined number of times to a guest terminal that has not entered by the elapse of a certain period of time. For example, when a conversation is possible, the operation server 130 may transmit information about the chat room and/or information about attending guests to the host terminal 111.

The operation server 130 may provide a conversation function by an avatar in operation 519 when all invited guest terminals 113 and 115 enter. In one embodiment, the operation server 130 may configure a conversation screen for each participant based on information collected from participating terminals 110 (e.g., host terminal 111 or guest terminals 113 and 115). . The operation server 130 may provide a conversation screen configured for each participant to the participating terminal 110. For example, the operation server 130 may change the conversation screen for each participant by considering events such as a change in speaker, manual manipulation of the participant, or movement of the participant. In order to change the conversation screen, the operation server 130 may collect information about events occurring in the participating terminals 110. In one embodiment, the operation server 130 may identify the speaker currently speaking based on the participant's voice data or an image taken of the participant. The operation server 130 may configure a new conversation screen for each participant by changing the angle and/or angle of view based on the identified speaker. In one embodiment, the operation server 130 may configure a new conversation screen to be provided to the participating terminals 110 by changing the angle and/or view angle in response to the manual operation of the participant (or user). In one embodiment, the operation server 130 acquires the movements of participants (e.g., host or guest) from images captured by the participating terminals 110, and refreshes the conversation screen for each participant reflecting the obtained movements. It can be configured. In one embodiment, the operation server 130 acquires the movements of participants (e.g., hosts or guests) based on sensing signals provided by the participating terminals 110, and creates a conversation for each participant reflecting the obtained movements. The screen can be reconfigured. The operation server 130 may provide information about the newly configured conversation screen to the participating terminals 110.

In operation 521, the operation server 130 may determine whether a request to end a conversation or a request to close a chat room is received from the participating terminal 110. The conversation termination request may be transmitted, for example, for the purpose of leaving the chat room in which the guest terminals 113 and 115 are currently participating. For example, the chat room closure request may be transmitted for the purpose of closing the chat room in which the host terminal 111 is currently participating.

When the operation server 130 receives an exit request from the guest terminal 113 or 115 in operation 523, it may allow the guest terminal 113 or 115 to leave the chat room. When the operation server 130 receives a request to close the chat room from the host terminal 111 in operation 523, it may close the chat room.

FIG. 6 is a signal flow diagram for performing an avatar conversation subroutine in a system (e.g., system 100 of FIG. 1), according to one embodiment. The signal flow shown in FIG. 6 is according to an embodiment in which the participating terminals 111 and 113 configure their conversation screen based on collected data (eg, voice, image, or sensing data).

Referring to FIG. 6, the first participating terminal (e.g., the first electronic device 111 in FIG. 1) using the avatar-based conversation function collects the user's voice data, image data, or sensing data in operation 610. can do. For example, the first participation terminal 111 may collect an electrical signal converted from the user's voice input through a microphone (e.g., the input module 1130 in FIG. 11) as voice data. For example, the first participating terminal 11 may collect an electrical signal corresponding to the user's image captured by a camera (eg, the camera module 1150 in FIG. 11) as image data. For example, the first participating terminal 111 may collect an electrical signal sensed by at least one sensor such as a gyroscope sensor (e.g., the sensor module 1160 of FIG. 11) as sensing data. The first participating terminal 111 may collect, for example, an electrical signal sensed by a wearable device equipped with a gyroscope sensor (e.g., the wearable device 120 of FIG. 1 or FIG. 11) as sensing data. there is. The collected voice data can be used, for example, to identify the speaker. The image data may, for example, identify the speaker, determine user movements (e.g., facial expressions, gaze, or body movements such as hands, arms, neck, or legs), or determine movement of the screen angle. It can be used to do this. The collected sensing data may be used, for example, to determine the user's movements (e.g., body movements such as hands, arms, neck, or legs) or movement of the screen angle.

In operation 620, the second participating terminal (e.g., the second electronic device 113 in FIG. 1) using the avatar-based conversation function together with the first participating terminal 111 sends the user's voice data, image data, or Sensing data can be collected. For example, the second participation terminal 113 may collect an electrical signal converted from the user's voice input through a microphone (e.g., the input module 1130 in FIG. 11) as voice data. For example, the second participating terminal 113 may collect an electrical signal corresponding to the user's image captured by a camera (eg, the camera module 1150 in FIG. 11) as image data. For example, the second participating terminal 113 may collect an electrical signal sensed by at least one sensor such as a gyroscope sensor (e.g., the sensor module 1160 of FIG. 11) as sensing data. The second participating terminal 113 may collect, for example, an electrical signal sensed by a wearable device equipped with a gyroscope sensor (e.g., the wearable device 120 of FIG. 1 or FIG. 11) as sensing data. there is. The collected voice data can be used, for example, to identify the speaker. The image data may, for example, identify the speaker, determine user movements (e.g., facial expressions, gaze, or body movements such as hands, arms, neck, or legs), or determine movement of the screen angle. It can be used to do this. The collected sensing data may be used, for example, to determine the user's movements (e.g., body movements such as hands, arms, neck, or legs) or movement of the screen angle.

The first participating terminal 111 may share the collected data with the second participating terminal 113 in operation 630. For example, the first participating terminal 111 may transmit at least one of collected voice data, image data, or sensing data to the second participating terminal 113. The second participating terminal 113 may share the collected data with the first participating terminal 111 in operation 630. For example, the second participating terminal 113 may transmit at least one of collected voice data, image data, or sensing data to the first participating terminal 111.

In operation 640, the first participating terminal 111 may configure a conversation screen based on at least one of data collected by the first participating terminal 111 and/or collected data provided from the second participating terminal 113. For example, the first participating terminal 111 may change the conversation screen in consideration of events such as speaker change or participant movement that can be confirmed through collected data. The first participation terminal 111 may identify the speaker currently speaking based on, for example, collected voice data of the participant or images taken of the participant. The first participating terminal 111 may configure a new conversation screen by changing the angle and/or angle of view based on the identified speaker. For example, the first participation terminal 111 acquires the movements of participants (e.g., hosts or guests) from images collected by the first participating terminal 111 or images taken by the second participating terminal 113, , a conversation screen that reflects the acquired movement can be newly configured. The first participating terminal 111 is, for example, based on the collected internal sensing signal or the external sensing signal provided by the second participating terminal 113 or the wearable device 120. The guest's movements can be acquired and a new conversation screen reflecting the acquired movements can be configured.

In operation 650, the second participating terminal 113 may configure a conversation screen based on at least one of data collected by the second participating terminal 113 and/or collected data provided by the first participating terminal 111. For example, the second participation terminal 113 may change the conversation screen in consideration of events such as speaker change or participant movement that can be confirmed through collected data. The second participation terminal 113 may identify the speaker currently speaking based on, for example, collected voice data of the participant or images taken of the participant. The second participating terminal 113 may configure a new conversation screen by changing the angle and/or angle of view based on the identified speaker. For example, the second participation terminal 113 acquires the movements of participants (e.g., hosts or guests) from images collected by the first participating terminal 111 or images taken by the first participating terminal 111. , a conversation screen that reflects the acquired movement can be newly configured. The second participating terminal 113, for example, connects participants (e.g., a host or The guest's movements can be acquired and a new conversation screen reflecting the acquired movements can be configured.

FIG. 7 is a signal flow diagram for performing an avatar conversation subroutine in a system (e.g., system 100 of FIG. 1), according to one embodiment. The signal flow shown in FIG. 7 displays a conversation screen of itself or another participating terminal (e.g., the second participating terminal 113) based on data (e.g., manual event data) collected by the first participating terminal 111. This is according to the constituting embodiment. In FIG. 7, it is assumed that manual event data is acquired by the first participating terminal 111, but the same procedure is performed even when manual event data is acquired by another participating terminal (e.g., the second participating terminal 113). Of course, the operation can be performed.

Referring to FIG. 7, the first participating terminal (e.g., the first electronic device 111 in FIG. 1) using the avatar-based conversation function may collect manual event data by the user's operation in operation 710. there is. For example, the first participation terminal 111 may collect information manually input by the user through a touch panel (eg, touch panel 1123 in FIG. 11) as manual event data. The manual event data can be used, for example, to determine the speaker. The manual event data can be used, for example, to determine the screen view (eg, zoom-in or zoom-out). The manual event data can be used, for example, to select an object to zoom in on. The passive event data may be used, for example, to determine the display format (e.g., landscape or portrait screen).

The first participating terminal 111 may transmit the collected passive event data to the second participating terminal 113 in operation 720.

In operation 730, the first participating terminal 111 may configure a conversation screen based on the manual event data collected by the first participating terminal 111. For example, the first participating terminal 111 may configure a new conversation screen by changing the angle and/or view angle based on the collected manual event data. For example, the first participating terminal 111 may identify the speaker currently speaking based on collected manual event data. For example, the first participating terminal 111 controls the angle, screen view (e.g., horizontal screen or vertical screen), and/or view angle (e.g., zoom-in or zoom-out) based on the identified speaker or designated target. ) can be changed to reconfigure the conversation screen. For example, the first participation terminal 111 may acquire the movements of participants (e.g., host or guest) based on collected manual event data and configure a new conversation screen reflecting the obtained movements. there is. For example, the first participation terminal 111 may acquire the movements of participants (e.g., host or guest) based on collected manual event data and configure a new conversation screen reflecting the obtained movements. there is.

In operation 740, the second participating terminal 113 may configure a conversation screen based on manual event data transmitted by the first participating terminal 111. For example, the second participating terminal 113 may configure a new conversation screen by changing the angle and/or view angle based on manual event data transmitted by the first participating terminal 111. For example, the second participating terminal 113 may identify the speaker currently speaking based on manual event data transmitted by the first participating terminal 111. The second participating terminal 113 changes the angle, screen view (e.g., horizontal screen or vertical screen), and/or angle of view (e.g., zoom-in or zoom-out) based on the identified speaker or designated target to conduct the conversation. The screen can be reconfigured. For example, the second participation terminal 113 acquires the movements of participants (e.g., host or guest) based on manual event data transmitted by the first participation terminal 111, and You can create a new conversation screen that reflects movement. For example, the second participation terminal 113 acquires the movements of participants (e.g., host or guest) based on manual event data transmitted by the first participation terminal 111, and You can create a new conversation screen that reflects movement.

FIG. 8 is a signal flow diagram for performing an avatar conversation subroutine in a system (e.g., system 100 of FIG. 1), according to one embodiment. The signal flow shown in FIG. 8 is a conversation for each participant in an operating server (e.g., server 130 in FIG. 1) based on data (e.g., voice, image, or sensing data) collected by participating terminals 111 and 113. This is according to an embodiment of configuring a screen (eg, a conversation screen of the first participating terminal 111 or a conversation screen of the second participating terminal 113).

Referring to FIG. 8, the first participating terminal (e.g., the first electronic device 111 in FIG. 1) using the avatar-based conversation function collects the user's voice data, image data, or sensing data in operation 810. can do. For example, the first participation terminal 111 may collect an electrical signal converted from the user's voice input through a microphone (e.g., the input module 1130 in FIG. 11) as voice data. For example, the first participating terminal 11 may collect an electrical signal corresponding to the user's image captured by a camera (eg, the camera module 1150 in FIG. 11) as image data. For example, the first participating terminal 111 may collect an electrical signal sensed by at least one sensor such as a gyroscope sensor (e.g., the sensor module 1160 of FIG. 11) as sensing data. The first participating terminal 111 may collect, for example, an electrical signal sensed by a wearable device equipped with a gyroscope sensor (e.g., the wearable device 120 of FIG. 1 or FIG. 11) as sensing data. there is.

The first participating terminal 111 may share the collected data with the operation server 130 in operation 830. For example, the first participation terminal 111 may transmit at least one of collected voice data, image data, or sensing data to the operation server 130.

In operation 820, the second participating terminal (e.g., the second electronic device 113 in FIG. 1) using the avatar-based conversation function together with the first participating terminal 111 generates the user's voice data, image data, or Sensing data can be collected. For example, the second participation terminal 113 may collect an electrical signal converted from the user's voice input through a microphone (e.g., the input module 1130 in FIG. 11) as voice data. For example, the second participating terminal 113 may collect an electrical signal corresponding to the user's image captured by a camera (eg, the camera module 1150 in FIG. 11) as image data. For example, the second participating terminal 113 may collect an electrical signal sensed by at least one sensor such as a gyroscope sensor (e.g., the sensor module 1160 of FIG. 11) as sensing data. The second participating terminal 113 may collect, for example, an electrical signal sensed by a wearable device equipped with a gyroscope sensor (e.g., the wearable device 120 of FIG. 1 or FIG. 11) as sensing data. there is.

The second participation terminal 113 may share the collected data with the operation server 130 in operation 840. For example, the second participation terminal 113 may transmit at least one of collected voice data, image data, or sensing data to the operation server 130.

In operation 850, the operation server 130 may configure a conversation screen for each participant based on collected data transmitted from the first participation terminal 111 and/or the second participation terminal 113. The conversation screen for each participant may be, for example, a first conversation screen for the first participation terminal 111 or a second conversation screen for the second participation terminal 113.

For example, the operation server 130 may change the conversation screen for each participant in consideration of events such as speaker change or participant movement that can be confirmed through the collected data. For example, the operation server 130 may identify the speaker currently speaking based on the participant's voice data included in the collected data or an image taken of the participant. The operation server 130 may configure a new conversation screen for each participant by changing the angle and/or angle of view based on the identified speaker. For example, the operation server 130 may acquire the movements of participants (e.g., host or guest) from image data included in the collected data, and configure a new conversation screen for each participant reflecting the obtained movements. You can. For example, the operation server 130 acquires the movements of participants (e.g., host or guest) based on the sensing signal included in the collected data, and refreshes the conversation screen for each participant reflecting the obtained movements. It can be configured.

The operation server 130 may transmit information (screen configuration data) about the first conversation screen configured for the first participating terminal 111 to the first participating terminal 111 in operation 860. The operation server 130 may transmit information (screen configuration data) about the second conversation screen configured for the second participating terminal 113 to the second participating terminal 113 in operation 870.

The first participation terminal 111 may configure the first conversation screen using information (screen configuration data) about the first conversation screen transmitted from the operation server 130 in operation 880. For example, the first participation terminal 111 displays a first conversation screen that has been changed in consideration of events such as speaker change or participant movement that can be confirmed through information (screen configuration data) about the first conversation screen. and the configured first conversation screen can be output through a display.

The second participation terminal 113 may configure a second conversation screen based on information (screen configuration data) about the second conversation screen transmitted from the operation server 130 in operation 890. For example, the second participation terminal 113 displays a second conversation screen that has been changed in consideration of events such as speaker change or participant movement that can be confirmed through information (screen configuration data) about the second conversation screen. and the configured second conversation screen can be output through a display.

FIG. 9 is a signal flow diagram for performing an avatar conversation subroutine in a system (e.g., system 100 of FIG. 1), according to one embodiment. The signal flow shown in FIG. 9 displays a conversation screen of itself or another participating terminal (e.g., the second participating terminal 113) based on data (e.g., manual event data) collected by the first participating terminal 111. This is according to the constituting embodiment. 9 assumes a situation in which manual event data is acquired by the first participating terminal 111, but the same procedure is performed even when manual event data is acquired by another participating terminal (e.g., the second participating terminal 113). Of course, the operation can be performed.

Referring to FIG. 9, the first participating terminal (e.g., the first electronic device 111 in FIG. 1) using the avatar-based conversation function may collect manual event data caused by the user's operation in operation 910. there is. For example, the first participation terminal 111 may collect information manually input by the user through a touch panel (eg, touch panel 1123 in FIG. 11) as manual event data. The manual event data can be used, for example, to determine the speaker. The manual event data can be used, for example, to determine the screen view (eg, zoom-in or zoom-out). The manual event data can be used, for example, to select an object to zoom in on. The passive event data may be used, for example, to determine the display format (e.g., landscape or portrait screen).

The first participation terminal 111 may transmit the collected manual event data to the operation server 130 in operation 920.

The operation server 130 may configure a conversation screen based on manual event data transmitted from the first participation terminal 111 in operation 930. For example, the operation server 130 may configure the first conversation screen of the first participation terminal 111 based on manual event data transmitted from the first participation terminal 111. For example, the operation server 130 may configure the second conversation screen of the second participation terminal 113 based on manual event data transmitted from the first participation terminal 111. For example, the operation server 130 may display the first conversation screen of the first participation terminal 111 and the second participation terminal 113 based on manual event data transmitted from the first participation terminal 111. ) can configure the second conversation screen.

For example, the operation server 130 may configure a new conversation screen by changing the angle and/or view angle based on the delivered manual event data. For example, the operation server 130 may identify the speaker currently speaking based on the delivered manual event data. For example, the operation server 130 adjusts the angle, screen view (e.g., horizontal screen or vertical screen), and/or view angle (e.g., zoom-in or zoom-out) based on the identified speaker or designated target. You can change the conversation screen to create a new one. For example, the operation server 130 may acquire the movements of participants (e.g., host or guest) based on the delivered manual event data and configure a new conversation screen reflecting the obtained movements. . For example, the operation server 130 may acquire the movements of participants (e.g., host or guest) based on the delivered manual event data and configure a new conversation screen reflecting the obtained movements. .

The operation server 130 may transmit information (screen configuration data) about the first conversation screen configured for the first participating terminal 111 to the first participating terminal 111 in operation 940. The operation server 130 may transmit information (screen configuration data) about the second conversation screen configured for the second participating terminal 113 to the second participating terminal 113 in operation 960.

The first participation terminal 111 may configure the first conversation screen using information (screen configuration data) about the first conversation screen transmitted from the operation server 130 in operation 950. For example, the first participation terminal 111 displays a first conversation screen that has been changed in consideration of events such as speaker change or participant movement that can be confirmed through information (screen configuration data) about the first conversation screen. and the configured first conversation screen can be output through a display.

The second participation terminal 113 may configure a second conversation screen based on information (screen configuration data) about the second conversation screen transmitted from the operation server 130 in operation 970. For example, the second participation terminal 113 displays a second conversation screen that has been changed in consideration of events such as speaker change or participant movement that can be confirmed through information (screen configuration data) about the second conversation screen. and the configured second conversation screen can be output through a display.

The above-described operations 960 and 970 may be selectively performed or omitted as needed.

FIG. 10 illustrates an avatar conversation function in an electronic device (e.g., the first or second participation terminals 111 and 113 of FIGS. 6 and 7 or the operation server 130 of FIGS. 8 and 9) according to an embodiment. This is a control flow chart for configuring the conversation screen. In one embodiment, the first participating terminal 111 may perform an operation according to the control flowchart of FIG. 10 in operation 640 of FIG. 6 or operation 730 of FIG. 7. In one embodiment, the second participating terminal 113 may perform an operation according to the control flowchart of FIG. 10 in operation 650 of FIG. 6 or operation 740 of FIG. 7. In one embodiment, the operation server 130 may perform an operation according to the control flowchart of FIG. 10 in operation 850 of FIG. 8 or operation 930 of FIG. 9.

Referring to FIG. 10 , the electronic device 110 or 130 may collect data for constructing a conversation screen in operation 1011. For example, the electronic device 110 or 130 may collect the user's voice data, image data, or sensing data.

The electronic device 110 can collect data to be referenced to construct a conversation screen using components included therein (e.g., microphone, camera, or sensor) or external devices (e.g., wearable device 120). there is. For example, the electronic device 110 may collect an electrical signal converted from the user's voice input through a microphone (e.g., the input module 1130 of FIG. 11) as voice data. For example, the electronic device 110 may collect an electrical signal corresponding to a user's image captured by a camera (eg, the camera module 1150 of FIG. 11) as image data. For example, the electronic device 110 may collect electrical signals sensed by at least one sensor, such as a gyroscope sensor (e.g., the sensor module 1160 of FIG. 11 ), as sensing data. For example, the electronic device 110 may collect electrical signals sensed by a wearable device equipped with a gyroscope sensor (e.g., the wearable device 120 of FIG. 1 or FIG. 11) as sensing data. The electronic device 130 may receive data collected by a participating terminal (e.g., the first and second participating terminals 111 and 113 in FIG. 8 or 9). The first and second participation terminals 111 and 113 may be, for example, electronic devices participating in a conversation using an avatar.

In operation 1013, the electronic devices 110 and 130 may determine whether a camera angle change event occurs. The camera angle change event may correspond to a situation that requires changing the angle applied to the current conversation screen to an angle in a different direction or from a different viewpoint. The camera angle change event may occur, for example, when the speaker changes or the object of interest changes due to manual manipulation by the user.

As an example, the electronic devices 110 and 130 may identify the speaker currently speaking based on the participant's voice data included in the collected data. For example, the electronic devices 110 and 130 may acquire the voice characteristics of the participants based on the participants' voice data included in the collected data. The electronic devices 110 and 130 may predict a participant having the acquired voice characteristic using information on reference voice characteristics preset for each participant. The electronic devices 110 and 130 may determine the predicted participant's avatar among the avatars participating in the conversation as the speaker avatar. The electronic devices 110 and 130 may collect voice data of attendees when the avatar-based conversation function is activated and conversations are conducted by attendees. The electronic devices 110 and 130 may obtain information about standard voice characteristics for each of the attendees using the collected voice data. The electronic devices 110 and 130 may set the information on the standard voice characteristic acquired for each participant as the standard voice characteristic corresponding to the participant, or may register it as the standard voice characteristic.

As an example, the electronic devices 110 and 130 may identify the speaker currently speaking based on the image captured of the participant included in the collected data. For example, the electronic devices 110 and 130 may obtain characteristic information about the mouth shape from the facial images of the participants. The electronic devices 110 and 130 can predict the speaker by determining whether the acquired mouth shape feature is a feature that can appear in the speaker. The electronic devices 110 and 130 may determine the avatar of the participant predicted to be the predicted speaker among the avatars participating in the conversation as the speaker avatar. The electronic devices 110 and 130 may collect features of the mouth shape that the speaker may have through repetitive learning to predict the speaker. The electronic devices 110 and 130 may set or store the collected mouth shape characteristics as reference information for predicting a speaker.

As an example, the electronic devices 110 and 130 may collect manual event data according to the user's manual operation and determine that an angle change event has occurred based on the collected manual event data.

When a camera angle change event occurs, the electronic devices 110 and 130 determine the angle to face the speaker avatar or a specific target specified by manual event data in operation 1025, and conduct a conversation based on the determined angle. You can configure the screen. The electronic devices 110 and 130 may output the configured conversation screen or transmit it to the electronic device to which it is to be applied. When constructing a conversation screen based on the determined angle, the electronic devices 110 and 130 may reflect the gaze or movement of the listener avatar based on the speaker avatar. For example, the electronic devices 110 and 130 may configure the conversation screen so that the gaze of the listener avatar is directed to the speaker avatar.

In operation 1015, the electronic devices 110 and 130 may determine whether a camera view angle change event occurs. The camera angle of view change event may correspond to a situation that requires changing the angle of view applied to the current conversation screen to a wider or narrower view angle. The camera view angle change event may occur, for example, when it corresponds to a zoom-out requirement or a zoom-in requirement, or by a user's manual operation (e.g., a user's swiping motion). The zoom-out requirement may occur, for example, when a participant enters for the first time and starts a conversation, when a situation arises where all participants must be checked, when participants speak at the same time, when not all participants speak, or when there is a short interval. This may include a situation where a conversation is exchanged. The zoom-in requirement may include, for example, situations in which a particular participant speaks at length in silent mode or a user selects a particular participant.

As an example, the electronic devices 110 and 130 may identify the speaker currently speaking based on the participant's voice data or image data captured by the participant included in the collected data. The method for the electronic devices 110 and 130 to determine a speaker based on voice data or image data can be applied in the same manner as described above. When the electronic device 110 or 130 determines a speaker, it zooms in or out by comprehensively considering the time when the speaker avatar opposing the speaker enters the chat room, the time the speaker is maintained as a speaker, or the presence of another speaker. You can determine whether you need to change the angle of view to apply out.

As an example, the electronic devices 110 and 130 may collect manual event data according to the user's manual operation (e.g., swiping) and determine that a view angle change event has occurred based on the collected manual event data. .

When a camera angle of view change event occurs, the electronic devices 110 and 130 change the angle of view to face the speaker avatar or a specific target specified by manual event data in operation 1027, and conduct a conversation based on the changed angle of view. You can configure the screen. The electronic devices 110 and 130 may output the configured conversation screen or transmit it to the electronic device to which it is to be applied. When constructing a conversation screen based on the changed viewing angle, the electronic devices 110 and 130 may reflect the gaze or movement of the listener avatar based on the speaker avatar. For example, the electronic devices 110 and 130 may configure the conversation screen so that the gaze of the listener avatar is directed to the speaker avatar.

The electronic devices 110 and 130 may determine whether a mirroring event occurs in operation 1017. The mirroring event may correspond to an event requesting that the user's actual facial expression or behavior, such as movement, be reflected on the user's avatar on the conversation screen.

As an example, the electronic devices 110 and 130 determine the occurrence of a mirroring event based on a change in facial expression or movement of a specific body part (e.g., neck, hand) from the user's image captured with the front camera. can do. The electronic devices 110 and 130 may determine the occurrence of a mirroring event based on a change in facial expression or movement of the participant based on data provided from another electronic device.

When a mirroring event occurs, the electronic devices 110 and 130 may configure a mirrored conversation screen in operation 1029. For example, the electronic devices 110 and 130 may check a change in the user's facial expression and reflect the confirmed facial expression change on the face of the corresponding avatar in the same or similar manner. As a similar reflection method, for example, a similar image can be selected from preset avatar images and the selected image can be applied to the conversation screen. For example, the electronic devices 110 and 130 may check the user's movement and reflect the confirmed movement in the same or similar way to the corresponding avatar. As a similar reflection method, for example, a similar image can be selected from preset avatar images and the selected image can be applied to the conversation screen. The electronic devices 110 and 130 may utilize sensing information provided from the wearable device 120 to improve accuracy for determining movement.

In operation 1019, the electronic devices 110 and 130 may determine whether a view type change event has occurred. The view type change event may include, for example, a situation in which the number of participants in a chat room changes. When a view type change event occurs, the electronic devices 110 and 130 may determine an appropriate view type for each situation and configure a conversation screen to which the determined view type is applied.

For example, when two people are having a conversation, the electronic devices 110 and 130 may configure and output a conversation screen with the other person's avatar placed near the center of the vertical display. At this time, for a sense of immersion, the electronic devices 110 and 130 may not display their avatars on the screen, but may process them transparently and maintain a constant viewing angle.

For example, when three people are having a conversation, the electronic devices 110 and 130 may output a conversation screen with opposing avatars arranged on both sides of the center of the vertical display. At this time, the electronic devices 110 and 130 do not display their own avatars on the screen, but process them transparently, and change their own avatar's viewpoint from the previous position to that of the other avatars so that the angle of view is wider than when two people are talking. It can be moved to a distant location.

For example, when four people are having a conversation, the electronic devices 110 and 130 may output a conversation screen with opposing avatars arranged near the center and on both sides on a horizontal display. At this time, as an example, the electronic devices 110 and 130 do not display the own avatar on the screen, but process it transparently, and move the viewpoint of the own avatar from the previous position to the other person so that the angle of view is wider than when three people are talking. It can be moved to a location away from the avatars.

As an example, when five people are having a conversation, the electronic devices 110 and 130 may output a conversation screen with opposing avatars arranged at substantially equal intervals near the center and on both sides on a horizontal display. At this time, as an example, the electronic devices 110 and 130 do not display their own avatars on the screen, but process them transparently, and move their avatars' viewpoints from the previous position to the other party so that the angle of view is wider than when four people are talking. It can be moved to a location away from the avatars.

For example, when six people are having a conversation, the electronic devices 110 and 130 may output a conversation screen in which the opponent's avatars are arranged at substantially equal intervals on a horizontal display. At this time, as an example, the electronic devices 110 and 130 do not display their own avatars on the screen, but process them transparently, and move their avatars' viewpoints from the previous position to the other party so that the angle of view is wider than when four people are talking. It can be moved to a location away from the avatars.

In operation 1021, the electronic devices 110 and 130 may determine whether a partner call event that calls the partner avatar occurs.

As an example, the electronic devices 110 and 130 may determine that a call event has occurred when the user touches the other person's avatar included in the conversation screen on the display. The electronic devices 110 and 130 may notify the conversation partner's electronic device that an avatar call has been requested based on the location where the user's touch was made and sensing information according to the touch. The other party's electronic device may output vibration or sound to notify the other party of the call.

As an example, the electronic devices 110 and 130 may determine whether the other party responded to the call by analyzing information provided from the other party's electronic device (e.g., a captured image of the other party). If it is determined that the other party has responded to the call, the electronic devices 110 and 130 may configure a conversation screen with the image changed so that the other party's avatar 2170 looks at the other party's avatar 2170.

In operation 1023, the electronic devices 110 and 130 may determine whether a recording request event occurs while an avatar-based conversation is being performed or after an avatar-based conversation has ended. When a recording request event occurs, the electronic devices 110 and 130 extract images corresponding to the conversation based on data collected during the conversation in operation 1035 and generate a conversation record image using the extracted images. You can. The electronic devices 110 and 130 may store the generated conversation record image or transmit it to an external electronic device.

FIG. 11 is a block diagram of an electronic device (eg, the electronic device 110 of FIG. 1 ) according to an embodiment.

Referring to FIG. 11, in one embodiment, the electronic device 110 includes a display module 1120 (e.g., display module 2560 of FIG. 23) and a processor 1110 (e.g., processor 2520 of FIG. 23). , input module 1130 (e.g., input module 2550 in FIG. 23), audio output module 1140 (e.g., audio output module 2555 in FIG. 23), camera module 1150 (e.g., in FIG. 23) Camera module 2580), sensor module 1160 (e.g., sensor module 2576 in FIG. 23), communication module 1170 (e.g., communication module 2590 in FIG. 23), or memory 1180 (e.g., It may include the memory 2530 of FIG. 23.

The display module 1120 may include a display panel 1121 or a touch panel 1123. The display panel 1121 can visually provide information to the outside of the electronic device 110 (eg, a user). The display module 1120 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. The touch panel 1123 may include a touch sensor configured to detect a touch by a user, or a pressure sensor configured to measure the intensity of force generated by the touch. The operation of the display module 1120 can be controlled by electrical connection with the processor 1110. The operation of the display panel 1121 or the touch panel 1123 included in the display module 1120 can be independently controlled by electrical connection with the processor 1110. In one embodiment, the display panel 1121 may visually provide display information to be displayed to the outside (eg, a user) under the control of the processor 1110.

The processor 1110 executes software (e.g., program 2540 in FIG. 23) to electrically connect the display module 1120, input module 1130, audio output module 1140, camera module 1150, and sensor. At least one other component (e.g., hardware or software component) such as the module 1160, the communication module 1170, or the memory 1180 may be controlled, and various data processing or operations may be performed. The processor 1110, as at least part of data processing or calculation, stores commands or data received from other components (e.g., display module 1120, communication module 1170, or input/output unit 127). It may be stored in 129 (e.g., volatile memory), or the command or data stored in the storage unit 129 may be processed and the processed result data may be stored in the storage unit 129.

The input module 1130 may receive commands or data to be used in a component of the electronic device 110 (e.g., the processor 1110) from outside the electronic device 110 (e.g., a user). The input module 1130 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 1140 may output sound signals to the outside of the electronic device 110. The sound output module 1140 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recorded playback. The receiver can be used to receive incoming calls. The receiver may be implemented separately from the speaker or as part of it.

The camera module 1150 can capture still images and moving images. The camera module 1150 may include one or more lenses, image sensors, image signal processors, or flashes.

The sensor module 1160 detects the operating state (e.g., power or temperature) of the electronic device 110 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can be created. The sensor module 1160 may include, for example, a gesture sensor, gyro sensor, barometric pressure sensor, magnetic sensor, acceleration sensor, grip sensor, proximity sensor, color sensor, IR (infrared) sensor, biometric sensor, temperature sensor, and humidity sensor. , or may include an illumination sensor.

The communication module 1170 may support establishment of a direct (eg, wired) communication channel or wireless communication channel with the wearable device 120, and communication through the established communication channel. The communication module 1170 operates independently of the processor 1110 and may include one or more communication processors that support direct (eg, wired) communication or wireless communication. The communication module 1170 is, for example, a wireless communication module (e.g., the wireless communication module 2592 of FIG. 23) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module. ) or a wired communication module (e.g., the wired communication module 2394 of FIG. 23) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module is a network 140 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct or infrared data association (IrDA), or a legacy cellular network, a 5G network, a next-generation communication network, the Internet, Alternatively, it may communicate with the wearable device 120, which is an external electronic device, through a computer network (e.g., a long-distance communication network such as LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips).

The memory 1180 is used by at least one component (e.g., display module 1120, processor 1110, communication module 1170, or input/output unit 127) of the electronic device 110. A variety of data can be stored. The data may include, for example, software (eg, program 2540) and input data or output data for commands related thereto. The storage unit 129 may include, for example, volatile memory or non-volatile memory.

The processor 1110 may include, for example, a face tracking module 1111, a hand tracking module 1113, or a motion tracking module 1115. The face tracking module 1111 may extract data of a part corresponding to the face from the user image captured by the camera module 1150 and track changes in facial expressions based on the extracted data. The hand tracking module 1113 may extract data of a part corresponding to the hand from the user image captured by the camera module 1150 and track hand movement based on the extracted data. The hand tracking module 1113 may track hand movement based on sensing information provided from the sensor module 1160 or sensing information provided from the wearable device 120. The motion tracking module 1115 may track the user's movement from the user image captured by the camera module 1150. The motion tracking module 1115 may track the user's movement based on sensing information provided from the sensor module 1160 or sensing information provided from the wearable device 120.

According to one embodiment, the processor 1110 may be configured to control the communication module 1170 to connect the wearable device 120 and then receive a request to provide display information. The processor 1110 may control the communication module 1170 to transmit some of the display information (eg, closed display information 435 in FIG. 4A) to the wearable device 120. The processor 1110 may be configured to control the display module 1120 to display some of the display information (e.g., open display information (e.g., open display information 433 in FIG. 4A)). 1110 may perform rendering on display information before displaying it on the display module 1120. The display information may be, for example, display information (full display information (full display information) to be displayed on the electronic device 110. For example, it may be full display information 431 in FIG. 4A) or partial display information (open display information (eg, open display information 433 in FIG. 4A)).

According to one embodiment, the processor 1110 may be configured to control the communication module 1170 to connect the wearable device 120 and then receive a request to provide display information. In response to the request, the processor 1110 transmits display information to the wearable device 120 through the communication module 1170, and selects some display information (open display information (e.g., open display information) to be displayed by the processor 1110 from the display information. The open display information 433 in Figure 4A may be separated. The processor 1110 may be configured to display some of the separated display information by controlling the display module 1120. The processor 1110 may perform rendering on some of the separated display information before displaying it on the display module 1120.

According to one embodiment, the processor 1110 may receive a layering change request from the wearable device 120 through the communication module 1170. In one embodiment, the processor 1110 may receive the layering change request from the wearable device 120. In response to the request, the processor 1110 may transmit display information (e.g., full display information 431 in FIG. 4A) to the wearable device 120 through the communication module 1170. In one embodiment, the processor 1110 may receive adjusted layering-related information from the wearable device 120 along with the layering change request. The processor 1110 obtains some display information (e.g., closed display information 435 in FIG. 4A) from display information using the layering-related information and transmits it to the wearable device 120 through the communication module 1170. Can be sent. The processor 1110 may be configured to obtain some display information (e.g., open display information 433 in FIG. 4A) from display information using the layering-related information and control the display module 1120 to display it. . The processor 1110 may perform rendering on display information to be displayed before displaying it on the display module 1120.

FIG. 12 is an example diagram of a user interface (UI) for each procedure for opening an avatar-based chat room in an electronic device (e.g., the electronic device 110 of FIG. 1), according to an embodiment.

Referring to FIG. 12, the display of the electronic device 110 may include an icon 1210 of an application program that can be selected to use an avatar-based conversation function (see (a)). The icon 1210 can be created on the screen of the display by installing an application program for using the avatar-based conversation function in the electronic device 110.

When the icon 1210 displayed on the screen of the display is selected, the electronic device 110 may output a user interface for selecting a guest to participate in an avatar-based conversation through the display (see (b)). For example, the electronic device 110 may output, through a display, a screen including a list of people who can be called for conversation and an icon 1220 that can request to start a conversation.

When the icon 1220 requesting to start a conversation is selected, the electronic device 110 displays the selection status 1240 of the guests to be called selected by the user from the target list displayed on the display and displays the icon 1230 for calling the guest. The included screen can be output through the display (see (c)).

When the icon 1230 for calling a guest is selected, the electronic device 110 can display a user interface 1250 for selecting a chat room background and an icon 1260 for requesting chat room entry through the display. There is (see (d)). When the icon 1260 that can request entry to the chat room is selected by the user, the electronic device 110 can enter the chat room in which the avatar-based conversation function can be used.

FIG. 13 illustrates an example of a user interface provided before entering a chat room for an avatar-based chat function in an electronic device (eg, the electronic device 110 of FIG. 1 ), according to one embodiment.

Referring to FIG. 13, the electronic device 110 corresponding to the host terminal for opening a chat room and calling a guest may display an avatar 1311 of an image waiting to enter on the display 1310 (see (a)). ).

When a request to participate in a conversation is received, the electronic device 110 corresponding to a guest terminal invited to an avatar-based conversation by the host terminal displays an avatar 1320 for requesting participation, an icon 1323 for agreeing to participate in the conversation, and An icon 1325 for refusing to agree to participate in a conversation may be displayed on the display 1320 (see (b)).

FIG. 14 illustrates an example of angle switching corresponding to a speaker change in an electronic device (e.g., the electronic device 110 of FIG. 1) according to an embodiment.

Referring to FIG. 14, when the user of the electronic device 110 is a speaker, the electronic device 110 includes a first avatar 1410, which is the speaker's own avatar, and second and third avatars, which are the listener avatars and the conversation partner. The first conversation screen facing (1420, 1430) can be displayed (see (a)).

When the speaker on the first conversation screen changes to the third avatar 1430 located to the right of the direction in which the first avatar 1410 is looking, the electronic device 110 changes the gaze of the first avatar 1410. By facing the third avatar 1430 and moving the angle to form the screen to the right, which is toward the third avatar 1430, a second conversation screen can be configured and output through the display (see (b)). ). In the second conversation screen, the angle may be adjusted so that the third avatar 1430, the speaker, is located near the center of the screen.

When the speaker on the second conversation screen changes to the second avatar 1420 located to the left of the direction in which the first avatar 1410 is looking, the electronic device 110 sends the first avatar 1410 and the third avatar 1410 to the second conversation screen. The gaze of the avatar 1430 is directed toward the second avatar 1420, and the angle to configure the screen is moved to the left, which is toward the second avatar 1420, to form a third conversation screen and output it through the display. (see (c)). In the third conversation screen, the angle may be adjusted so that the second avatar 1420, the speaker, is located near the center of the screen.

If both the second avatar 1420 and the third avatar 1430 located to the left and right in the direction in which the first avatar 1410 is looking in the third conversation screen are speakers, the electronic device 110 The gaze of the first avatar 1410 is directed to the front, the second avatar 1420 and the third avatar 1430 face each other, and the angle at which the screen is composed is directed to the front of the first avatar 1410. By moving it, a fourth conversation screen can be created and output through the display (see (d)). In the fourth conversation screen, the angles of the speakers, the second avatar 1420 and the third avatar 1430, may be adjusted so that they are substantially symmetrical about the center on both sides of the screen.

As shown, by adjusting the listener's viewpoint and the angle of the screen in the avatar-based conversation screen centered on the speaker, it is possible to provide the user with a more realistic conversation experience in a virtual space. For example, just by the user holding the electronic device 110 without requiring any manipulation from the user, a screen can be provided in which the face of the avatar on behalf of the user is turned to face the speaker avatar.

FIGS. 15A and 15B illustrate an example of angle switching corresponding to manual manipulation in an electronic device (eg, the electronic device 110 of FIG. 1 ), according to an embodiment.

Referring to FIG. 15A, the electronic device 110 may display an icon 1530 for manually controlling angle switching on a conversation screen displayed on the display (see (a)). Before the icon 1530 is manipulated by the user, the electronic device 110 displays a conversation screen configured so that the first avatar 1510 and the second avatar 1520, which are the opposing avatars participating in the conversation, face each other from both sides. can be output through the display.

When the icon 1530 is swiped to the left by the user (1540), the electronic device 110 moves the angle of the screen to the left and the first avatar 1510 moves to the right and is positioned near the center. In addition, the second avatar 1520 can also change the conversation screen to move to the right and output it through the display (see (b)).

When the icon 1530 is swiped to the right by the user (1550), the electronic device 110 moves the angle of the screen to the right and the second avatar 1520 moves to the left and is positioned near the center. In addition, the conversation screen can be changed to move the first avatar 1510 to the left and output through the display (see (c)).

Referring to FIG. 15B, the electronic device 110 may display a conversation screen including icons 1560, 1570a, 1570b, 1580a, and 1580b for manually controlling angle switching on the display. The icons 1560, 1570, and 1580 that the user can manually manipulate to change the angle include, for example, a main indicator 1560 indicating the user's status or at least one sub indicator indicating the other party's status. May include characters (1570, 1580). As an example, the main indicator 1560 has a semicircular shape and may respond booming in response to the voice generated by the utterance. The booming reaction may correspond to a reaction in which the semicircle shape gradually spreads. As an example, at least one sub indicator 1570 or 1580 may be toggled by a user's touch. The color or brightness of the at least one sub indicator 1570 or 1580 may change in response to the operating state according to the toggle operation. The sub indicators 1570 and 1580 may be generated by reflecting the number of people participating in the conversation or their positions on the screen. States that can be identified by the sub-indicators 1570 and 1580 may include, for example, first display states 1570a and 1580a indicating an operating state that automatically changes the screen according to the speaker. The state that can be identified by the sub indicators (1570a, 1570b, 1580a, 1580b) is, for example, an operating state in which manual control is possible so that the screen is focused on the opponent's avatar (1510, 1520) desired by the user. It may include second display states 1570b and 1580b. For example, the first display states 1570a and 1580a and the second display states 1570b and 1580b may be alternately switched in proportion to the number of times the user taps the sub indicators 1570 and 1580.

As an example, (a) shows a situation in which sub-indicators 1570 and 1580 are displayed in first display states 1570a and 1580a. In the first display states 1570a and 1580a, the electronic device 110 may automatically change the screen according to the speaker. For example, the electronic device 110 may provide a screen in which the user's avatar changes the other person's avatars 1510 and 1520 in proportion to the swiping distance.

For example, in (b), the first sub indicator 1570 corresponding to the first counterpart avatar 1510 is displayed in the second display state 1570b, and the second sub indicator 1570 corresponding to the second counterpart avatar 1520 is displayed. It shows a situation where the indicator 1580 is displayed as the first display state 1580a. Since the first sub-indicator 1570 is in the second display state 1570b, the electronic device 110 can provide a screen with the center or focus of the screen on the first counterpart's avatar 1510.

For example, in (c), the first sub indicator 1570 corresponding to the first counterpart avatar 1510 is displayed in the first display state 1570a, and the second sub indicator 1570 corresponding to the second counterpart avatar 1520 is displayed in the first display state 1570a. This shows a situation where the indicator 1580 is displayed in the second display state 1580b. Since the second sub indicator 1580 is in the second display state 1580b, the electronic device 110 can provide a screen with the center or focus of the screen on the second counterpart's avatar 1520.

As shown, by adjusting the angle of the avatar-based conversation screen in response to the user's manual manipulation, it is possible to provide the user with a more realistic conversation experience in a virtual space.

FIG. 16 illustrates an example of switching the angle of view in response to setting requirements in an electronic device (e.g., the electronic device 110 of FIG. 1), according to an embodiment.

Referring to FIG. 16, the electronic device 110 may provide a conversation screen (a) in a zoomed-out view or a conversation screen (b) in a zoomed-in view using preset requirements.

For example, when a participant first enters a chat room and starts a conversation, the participant's electronic device 110 may output a conversation screen (a) in a zoomed-out view through the display. For example, when a participant performs a manual operation to check all conversation participants, the participant's electronic device 110 may output a conversation screen (a) in a zoomed-out view through the display. For example, when all participants in a conversation speak at the same time or when no one speaks at the same time, the participant's electronic device 110 may output a conversation screen (a) in a zoomed-out view through the display. For example, when participants participating in a conversation exchange a conversation at a short interval (e.g., 3 seconds), the participant's electronic device 110 may output a conversation screen (a) in a zoomed-out view through the display. For example, when a specific participant speaks for longer than a preset time, the electronic device 110 may output a conversation screen (b) providing a zoomed-in view of the specific participant's avatar through the display. For example, when a participant manually selects a specific participant, the participant's electronic device 110 may output a conversation screen (b) providing a zoomed-in view of the specific participant's avatar through the display.

According to the above, the zoom-out requirement for providing a zoomed-out view on the conversation screen is when a participant first enters a chat room and starts a conversation, when a participant performs a manual operation to check all conversation participants, or when a participant enters a conversation room for the first time and starts a conversation, or when a participant performs a manual operation to check all conversation participants, or all participants in the conversation are This can include situations where participants are speaking simultaneously, where no one is speaking, or where participants are speaking at short intervals (e.g., 3 seconds). Zoom-in requirements that provide a zoom-in view of the conversation screen may include situations where a specific participant speaks longer than a preset time or where a specific participant is manually selected.

FIG. 17 illustrates an example of switching a view (angle of view) in response to a manual manipulation in an electronic device (eg, the electronic device 110 of FIG. 1 ), according to an embodiment.

Referring to FIG. 17, the electronic device 110 pre-establishes a swipe pattern for manually manipulating view switching and provides a conversation screen in which the view is switched in response to the swipe pattern manually entered by the user. You can.

For example, when the user 1740 inputs a swipe pattern moving from right to left on the first conversation screen where the first avatar 1710 and the second avatar 1720 are talking while facing each other, the electronic device 110 By moving the camera angle to the right, a second conversation screen providing a zoomed-in view of the second avatar 1720 can be output through the display.

For example, when the user 1740 inputs a swipe pattern moving from left to right on the second conversation screen that provides a zoomed-in view of the second avatar 1720, the electronic device 110 changes the angle of the camera to the left. While moving, a third conversation screen providing a zoomed-in view of the first avatar 1710 can be output through the display.

For example, when the user 1740 inputs a swipe pattern moving from the bottom to the top in the third conversation screen that provides a zoomed-in view of the first avatar 1710, the electronic device 110 changes the camera angle to two. While moving near the center of the avatars 1710 and 1720, a fourth conversation screen providing a zoomed-out view of the first and second avatars 1710 and 1720 may be output through the display.

The user's image 1730 captured by the camera may be displayed in the upper left corner of the first to fourth conversation screens displayed on the display of the electronic device 110.

According to the above, the user can be provided with a conversation screen with a desired view by inputting a simple preset swipe pattern on the display.

FIGS. 18A and 18B illustrate an example of mirroring a user's movements on an avatar using an image captured by a camera in an electronic device (e.g., the electronic device 110 of FIG. 1 ), according to an embodiment.

Referring to FIG. 18A, the electronic device 110 captures an image including the user's face 1830 through the front camera 1820 and applies a face tracking function to analyze the expression of the face 1830 in the captured image. can do. The electronic device 110 may express the analyzed expression by mirroring the facial expression of the avatar 1840 displayed on the screen of the display 1810.

Referring to FIG. 18B, the electronic device 110 captures an image including the user's hand 1850 through the front camera 1820 and applies a hand tracking function to analyze the movement of the hand 1850 in the captured image. can do. The electronic device 110 may express the analyzed movement by mirroring the hand movement of the avatar 1870 displayed on the screen of the display 1810. The electronic device 110 may receive an image of a participant participating in a chat room captured through a camera by the electronic device of the participant, and analyze the movement of the hand 1850 in the received image. The electronic device 110 may express the analyzed movement by mirroring the hand movement of the other avatar 1860 displayed on the screen of the display 1810.

The electronic device 110 may additionally consider sensing information provided from a wearable device (e.g., wearable device 120 of FIG. 1) in order to accurately analyze the user's movements. As an example, the electronic device 110 receives sensing information detected by at least one sensor (e.g., a gyroscope sensor) included in a smart watch, which is one of the wearable devices 120, and accurately analyzes the user's hand movements. You can.

As described above, the electronic device 110 acquires a facial expression or gesture from the user's image captured by the front camera, and mirrors the acquired facial expression or gesture to the user's avatar included in the conversation screen. It can be expressed by mirroring.

As described above, the electronic device 110 acquires the facial expressions or gestures of the attendees from the collected data, and mirrors the acquired facial expressions or gestures of the attendees onto the attendees' avatars included in the conversation screen. It can be expressed by mirroring.

FIGS. 19A, 19B, and 19C show examples of movements that are preset in an electronic device (e.g., the electronic device 110 of FIG. 1) and can be applied to an avatar, according to an embodiment.

Referring to FIG. 19B, the electronic device 110 displays the movements of avatars 1910 and 1920 that are expected to be frequently used by users (e.g., (a) joy (1951), (b) anger (1953), and (c) love. Expression (1955)) can be set in advance.

As an example, (a) shows a conversation screen in which a first avatar 1910 and a second avatar 1920 are participating on behalf of two participants. The conversation screen may include, for example, a function key 1930 for activating icons 1951, 1953, and 1955 for expressing one's status. The function key 1930 can be manipulated by the user to convey his or her emotions using a preset avatar gesture. The user's operation may be performed, for example, by touching the function key 1930.

For example, in (b), when the user manipulates the function key 1930, the electronic device 110 may display icons 1951, 1953, and 1955 for selecting a preset avatar gesture. The user can select one of the displayed icons (1951, 1953, and 1955). When an icon is selected by a user, the electronic device 110 may transmit an image of an avatar making a preset gesture to express an emotion corresponding to the selected icon to the other participant's electronic device. In this case, an image of an avatar on behalf of the user making the selected gesture may be displayed on the other participant's electronic device.

Referring to FIG. 19B, the electronic device 110 can preset avatar movements (e.g., (a) thinking, (b) laughing, (c) cheering, and (d) expressing love) that are expected to be frequently used by users. there is. The electronic device 110 may analyze the user's manual operation or captured image or voice and mirror a motion corresponding to one of preset movements to the avatar.

Referring to FIG. 19C, the electronic device 110 may obtain sensing information informing the user's status from an external electronic device such as a wearable device that can be connected through a wireless channel. The electronic device 110 can predict the user's physical state or the action being taken based on sensing information. The electronic device 110 may express the user's predicted physical state or motion as a motion of an avatar included on behalf of the user on the conversation screen.

For example, when the electronic device 110 recognizes that the user is in a sleeping state based on sensing information provided from a smart watch, which is one of the wearable devices, the avatar 1990 included in the conversation screen makes a gesture expressing fatigue on behalf of the user. It can be indicated to be taken. Additionally, the electronic device 110 may transmit information indicating the user's status to the other participant's electronic device based on the network environment. In this case, the other participant's electronic device may display an avatar representing the user making a gesture expressing fatigue.

FIGS. 20A to 20E illustrate an example of changing the screen shape in consideration of the number of participants in an electronic device (e.g., the electronic device 110 of FIG. 1 ), according to an embodiment.

Referring to FIG. 20A, when two people are having a conversation, the electronic device 2010 may output a conversation screen with the opponent's avatar 2032 placed near the center of the display 2020, which has a vertical screen shape. At this time, for immersion, the electronic device 2010 may not display its own avatar 2031 on the screen, but may render it transparent and maintain a constant viewing angle.

Referring to FIG. 20b, when three people are having a conversation, the electronic device 2010 outputs a conversation screen with opposing avatars 2042 and 2043 arranged on both sides of the center of the display 2020, which has a vertical screen shape. You can. At this time, the electronic device 2010 does not display the own avatar on the screen, but processes it transparently, and moves the viewpoint of the own avatar from the previous position (2041-1) to the other avatar so that the angle of view is wider than when two people are talking. It can be moved to a location (2041-2) away from the fields (2042, 2043).

Referring to FIG. 20C, when four people are having a conversation, the electronic device 2010 displays a conversation screen with opposing avatars 2052, 2053, and 2054 arranged near the center and on both sides of the display 2020, which has a horizontal screen shape. can be output. At this time, the electronic device 2010 does not display the own avatar on the screen, but processes it transparently, and moves the viewpoint of the own avatar from the previous position (2051-1) to the other avatar so that the angle of view is wider than when three people are talking. It can be moved to a location (2051-2) away from the field (2052, 2053, 2054).

Referring to FIG. 20D, when five people are having a conversation, the electronic device 2010 displays opposing avatars 2062, 2063, 2064, and 2065 near the center and on both sides of the display 2020, which has a horizontal screen. You can output conversation screens arranged at even intervals. At this time, the electronic device 2010 does not display the own avatar on the screen, but processes it transparently, and moves the viewpoint of the own avatar from the previous position (2061-1) to the other avatar so that the angle of view is wider than when four people are talking. It can be moved to a location (2061-2) away from the fields (2062, 2063, 2064, 2065).

Referring to FIG. 20e, when six people are having a conversation, the electronic device 2010 displays the opposing avatars 2072, 2073, 2074, 2075, and 2076 at substantially equal intervals on the display 2020 with a horizontal screen shape. You can print the placed conversation screen. At this time, the electronic device 2010 does not display the own avatar on the screen, but makes it transparent, and moves the viewpoint of the own avatar from the previous position (2071-1) to the other avatar so that the angle of view is wider than when four people are talking. It can be moved to a location (2071-2) away from fields (2072, 2073, 2074, 2075, 2076).

As described above, the electronic device 2010 may improve the angle of view by considering the number of avatars participating in the conversation (or the number of attendees). The upper limb electronic device 110 may output a conversation screen prepared by the adjusted viewing angle. That is, the electronic device 110 can change the shape of the screen from vertical to horizontal in proportion to an increase in the number of avatars participating in the conversation, or move the viewpoint to widen the angle of view.

FIG. 21 illustrates an example of calling the other avatar through manual manipulation in an electronic device (e.g., the electronic device 110 of FIG. 1), according to an embodiment.

Referring to FIG. 21, the electronic device 110 may output a conversation screen on the display 2110. For example, the conversation screen may include the opponent's avatar 2130 (see (a)). For example, the conversation screen may display a user's image 2120 captured through a camera at the upper left corner of the display 2110. The user may use the finger 2140 to touch (2150) the opponent's avatar (2130) included in the conversation screen. The electronic device 110 may transmit a request for an avatar call to the electronic device of the conversation partner based on the location where the user's touch 2150 was made and the sensing information according to the touch 2150 (2160). The other party's electronic device may output vibration or sound to notify the other party of the call.

The electronic device 110 may determine whether the other party responded to the call by analyzing information provided from the other party's electronic device (e.g., a captured image of the other party). If the electronic device 110 determines that the other party has responded to the call, it can change and display the image on the display 2110 so that the other party's avatar 2170 is looking at the other party (see (b)).

As described above, when the electronic device 110 detects that the user touches the conversation screen, it can receive a signal to call an external electronic device corresponding to the touched avatar.

FIG. 22 illustrates an example of recording a conversation screen at the request of a user in an electronic device (e.g., the electronic device 110 of FIG. 1), according to an embodiment.

Referring to FIG. 21, while an avatar-based conversation is in progress or after an avatar-based conversation has ended, the electronic device 110 responds to a user's request by displaying images corresponding to the conversation based on data collected during the conversation. It can be extracted. The electronic device 110 may generate a conversation record image using the extracted images. The electronic device 110 may store the generated conversation record image or transmit it to an external electronic device.

FIG. 23 is a signal flow diagram for processing a gift event during a conversation using an avatar in a system (e.g., system 100 of FIG. 1), according to one embodiment.

Referring to FIG. 23, a first participating terminal (e.g., the first participating terminal 111 in FIG. 1) connects with at least one counterpart terminal (e.g., the second participating terminal 113 in FIG. 1) in operation 2301. You can conduct conversations using avatars. The detailed operation of conducting a conversation using the avatar is as described above. In the drawing, it is assumed that two people participate in the conversation, but of course, the operations described later can be applied in the same or similar manner to situations where more participants participate. For example, in Figure 24a, three participants (e.g., the first participant avatar 2410, the second participant avatar 24220, and the third participant avatar 2430 in Figure 24A) are having a conversation in an established chat room. A conversation screen of the situation is shown.

In operation 2303, the first participating terminal 111 may display a menu window on the display in response to the user's request. For example, the menu window may include items that allow the user to select functions that can be used during a conversation using an avatar. As an example, Figure 24b shows a conversation screen displaying a menu window 2450 containing items corresponding to functions such as gifting, image sharing, gesture addition, item shop, and exit.

The first participating terminal 111 can check the item selected by the user in the menu window 2450 displayed on the conversation screen. In operation 2303, the first participating terminal 111 may recognize that the user has selected the 'Gift' item from among the items included in the menu window 2450. When the user selects the 'Gift' item among the menu items, the first participation terminal 111 may display a screen where the user can select a gift. As an example, (a) of FIG. 24C shows a coupon shovel screen where you can select a coupon as a gift.

When the user selects the 'Gift' item among the menu items, the first participation terminal 111 may perform an operation of selecting a product to be gifted and a recipient to receive the selected product in operation 2307. The user can select a product to gift from among the products displayed on the screen. As an example, in (a) of FIG. 24C, it is assumed that the coffee coupon 2461 is selected by the user. When a product to be gifted is selected by the user, the first participation terminal 111 can output detailed information about the selected product on the screen. As an example, in (b) of FIG. 24C, detailed information about the selected coffee coupon (e.g., price information, product description, etc.) and a selection button to finally select an event item to process the corresponding product (e.g., purchase (2463) ) or gift (2465)).

When the user completes selecting a product to be gifted, the first participation terminal 111 performs an operation to allow the user to select a recipient (e.g., at least one of the participants participating in the conversation) to whom the product is to be delivered. can do. As an example, Figure 24D shows a screen displaying a 'Send' button 2470 to confirm whether to send a gift to an image of the user's avatar 2410 holding a gift box.

The first participation terminal 111 may request the user to select a recipient to whom the gift will be delivered when the user touches or presses the 'Send' button 2470. As an example, Figure 24e shows a screen displaying selection buttons 2481 and 2483 that can be selected as recipients above the avatars 2420 and 2430 of participants participating in the conversation. When one of the selection buttons 2481 and 2483 displayed on the screen is touched or pressed by the user, the first participation terminal 111 recognizes this as selecting the recipient and selects the avatar or participant corresponding to the touched or pressed selection button to the recipient. You can select .

When a product to be gifted and a recipient are selected, the first participation terminal 111 may display a screen in which an avatar 2410 on behalf of the user sends a gift in operation 2309. As an example, Figure 24f shows a screen depicting a gift box leaving the hand of avatar 2410. The screen showing the gift box leaving the hand of the avatar 2410 may be provided as a photo or video. As an example, the screen shown in FIG. 26f may also be displayed through the display of the second participating terminal 113, which is the other participating terminal that will receive the gift. As an example, the screen shown in FIG. 26f can also be displayed through the display of a participating terminal other than the other participating terminal that will receive the gift. Whether the image depicting the scene where the gift is sent may be displayed on the display of the second participating terminal 113, which is the other participating terminal that will receive the gift, or another participating terminal, may be determined by the settings of the user sending the gift. Meanwhile, although not shown, in order to display an image depicting a scene in which a gift is sent on the display of another participating terminal, an operation for the first participating terminal 111 to transmit related information to the participating terminal must be added. There is a need.

In operation 2311, the first participating terminal 111 may transmit gift information for gifting the selected product to the second participating terminal 113 corresponding to the recipient of the gift. The gift information may include, for example, information about the product corresponding to the gift or information about the person who sent the gift.

The second participating terminal 113 may receive gift information transmitted by the first participating terminal 111 in operation 2311. When receiving the product information, the second participation terminal 113 may display a screen where an avatar 2430 on behalf of the user receives the gift. As an example, Figure 24g shows a screen depicting a gift box falling into the hand of the avatar 2430. A screen showing a gift box falling into the hand of the avatar 2430 may be provided as a photo or video. As an example, the screen shown in FIG. 26g may also be displayed through the display of the first participating terminal 111, which is the participating terminal that delivered the gift. As an example, the screen shown in FIG. 26g may also be displayed through the display of a participating terminal other than the second participating terminal 113 that received the gift. Whether or not the image depicting the scene of receiving the gift may be displayed on a display of a participating terminal other than the second participating terminal 113 may be determined by the settings of the user sending the gift. As an example, the screen shown in FIG. 26g is displayed on the display of a participating terminal other than the first participating terminal 111 or the second participating terminal 113 when the second participating terminal 113 receives the gift without rejecting it. It could be.

The second participating terminal 113 can perform an operation to select whether to accept or reject a gift. For example, in Figure 24h, the image of the avatar 2430 of the second participating terminal 113 holding a gift box shows a 'receive' button 2491 to receive a gift and/or a 'reject' button to reject the gift. A screen displaying a button 2493 is shown.

When the user touches or presses the 'Receive' button 2491, the second participation terminal 113 can display information about the product or information about the person who gave the gift using the received gift information. As an example, Figure 24i shows a screen displaying my 'coupon box' including a 'coffee coupon' received as a gift.

When the user touches or presses the 'Receive' button 2491, the second participating terminal 113 may transmit a message confirming that the gift has been delivered to the first participating terminal 111 in operation 2315.

The first participating terminal 111 may receive a gift delivery confirmation message from the second participating terminal 113 in operation 2315. When receiving the gift delivery confirmation message, the first participation terminal 111 may display a screen in which the other party's avatar 2430 receives the gift in operation 2317. For example, the screen output from the first participating terminal 111 may be the same screen as in FIG. 24G.

Figures 24A to 24I are exemplary diagrams of a user interface (UI) for each procedure for delivering a gift on a conversation screen using an avatar, according to an embodiment.

FIG. 24A shows an example of a conversation screen of a chat room in which three participants (e.g., the first participant avatar 2410, the second participant avatar 24220, and the third participant avatar 2430 in FIG. 24A) are opened. I'm doing it.

FIG. 24B shows an example of a conversation screen in which a menu window 2450 is displayed including items corresponding to functions such as gifting, image sharing, gesture addition, item shop, and exit.

In (a) of Figure 24c, an example of a screen representing a coupon shovel selling coupons such as a coffee coupon 2461 that can be selected for a gift is shown, and in (b) of Figure 24c, the selected coffee coupon ( 2661) A screen containing detailed information (e.g., price information, product description, etc.) and a selection button (e.g., purchase (2463) or gift (2465)) to finally select an event item to process the product. An example is shown.

FIG. 24d shows an example of a screen displaying a 'Send' button 2470 to confirm whether to send a gift to an image in which the user's avatar 2410 is holding a gift box.

FIG. 24e shows an example of a screen displaying selection buttons 2481 and 2483 for selecting recipients above the avatars 2420 and 2430 of participants participating in a conversation.

Figure 24f shows an example of a screen depicting a gift box leaving the hand of avatar 2410.

Figure 24g shows an example of a screen depicting a gift box falling into the hand of an avatar 2430.

In Figure 24h, the avatar 2430 of the second participating terminal (e.g., the second participating terminal 113 in Figure 1) is shown in an image holding a gift box and a 'receive' button 2491 to receive a gift and/or a gift. An example of a screen displaying a 'reject' button 2493 for rejecting is shown.

Figure 24i shows an example of a screen displaying my 'coupon box' including a 'coffee coupon' received as a gift.

FIG. 25 is a block diagram of an electronic device 2501 (eg, electronic device 110 of FIG. 1 ) in a network environment 2500, according to various embodiments.

Referring to FIG. 25, in the network environment 2500, the electronic device 2501 communicates with the electronic device 2502 through a first network 2598 (e.g., a short-range wireless communication network) or a second network 2599. It is possible to communicate with at least one of the electronic device 2504 or the server 2508 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 2501 may communicate with the electronic device 2504 through the server 2508. According to one embodiment, the electronic device 2501 includes a processor 2520, a memory 2530, an input module 2550, an audio output module 2555, a display module 2560, an audio module 2570, and a sensor module ( 2576), interface (2577), connection terminal (2578), haptic module (2579), camera module (2580), power management module (2588), battery (2589), communication module (2590), subscriber identification module (2596) , or may include an antenna module 2597. In some embodiments, at least one of these components (eg, the connection terminal 2578) may be omitted, or one or more other components may be added to the electronic device 2501. In some embodiments, some of these components (e.g., sensor module 2576, camera module 2580, or antenna module 2597) are integrated into one component (e.g., display module 2560). It can be.

The processor 2520, for example, executes software (e.g., program 2540) to operate at least one other component (e.g., hardware or software component) of the electronic device 2501 connected to the processor 2520. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of the data processing or computation, the processor 2520 stores instructions or data received from another component (e.g., the sensor module 2576 or the communication module 2590) in the volatile memory 2532. The commands or data stored in the volatile memory 2532 can be processed, and the resulting data can be stored in the non-volatile memory 2534. According to one embodiment, the processor 2520 may include a main processor 2521 (e.g., a central processing unit or an application processor) or an auxiliary processor 2523 (e.g., a graphics processing unit, a neural network processing unit) that can operate independently or together with the main processor 2521. It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 2501 includes a main processor 2521 and a auxiliary processor 2523, the auxiliary processor 2523 may be set to use lower power than the main processor 2521 or be specialized for a designated function. You can. The auxiliary processor 2523 may be implemented separately from the main processor 2521 or as part of it.

The auxiliary processor 2523 may, for example, act on behalf of the main processor 2521 while the main processor 2521 is in an inactive (e.g., sleep) state, or while the main processor 2521 is in an active (e.g., application execution) state. ), together with the main processor 2521, at least one of the components of the electronic device 2501 (e.g., the display module 2560, the sensor module 2576, or the communication module 2590) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 2523 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 2580 or communication module 2590). there is. According to one embodiment, the auxiliary processor 2523 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. This learning may be performed, for example, in the electronic device 2501 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 2508). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 2530 may store various data used by at least one component (eg, the processor 2520 or the sensor module 2576) of the electronic device 2501. Data may include, for example, input data or output data for software (e.g., program 2540) and instructions related thereto. Memory 2530 may include volatile memory 2532 or non-volatile memory 2534.

The program 2540 may be stored as software in the memory 2530 and may include, for example, an operating system 2542, middleware 2544, or applications 2546.

The input module 2550 may receive commands or data to be used in a component of the electronic device 2501 (e.g., the processor 2520) from outside the electronic device 2501 (e.g., a user). The input module 2550 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 2555 can output sound signals to the outside of the electronic device 2501. The sound output module 2555 may include, for example, a speaker or receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 2560 can visually provide information to the outside of the electronic device 2501 (eg, a user). The display module 2560 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 2560 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 2570 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 2570 acquires sound through the input module 2550, the sound output module 2555, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 2501). Sound may be output through an electronic device 2502 (e.g., speaker or headphone).

The sensor module 2576 detects the operating state (e.g., power or temperature) of the electronic device 2501 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 2576 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 2577 may support one or more designated protocols that can be used to connect the electronic device 2501 directly or wirelessly with an external electronic device (e.g., the electronic device 2502). According to one embodiment, the interface 2577 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 2578 may include a connector through which the electronic device 2501 can be physically connected to an external electronic device (eg, the electronic device 2502). According to one embodiment, the connection terminal 2578 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 2579 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 2579 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 2580 can capture still images and videos. According to one embodiment, the camera module 2580 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 2588 can manage power supplied to the electronic device 2501. According to one embodiment, the power management module 2588 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

Battery 2589 may supply power to at least one component of electronic device 2501. According to one embodiment, the battery 2589 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

Communication module 2590 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between the electronic device 2501 and an external electronic device (e.g., electronic device 2502, electronic device 2504, or server 2508). It can support establishment and communication through established communication channels. Communication module 2590 operates independently of processor 2520 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 2590 may be a wireless communication module 2592 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 2394 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 2598 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 2599 (e.g., legacy It may communicate with an external electronic device 2504 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 2592 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 2596 to communicate within a communication network such as the first network 2598 or the second network 2599. The electronic device 2501 can be confirmed or authenticated.

The wireless communication module 2592 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 2592 may support high frequency bands (e.g., mmWave bands), for example, to achieve high data rates. The wireless communication module 2592 uses various technologies to secure performance in high frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 2592 may support various requirements specified in the electronic device 2501, an external electronic device (e.g., electronic device 2504), or a network system (e.g., second network 2599). According to one embodiment, the wireless communication module 2592 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 2597 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module 2597 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 2597 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for the communication method used in the communication network, such as the first network 2598 or the second network 2599, is connected to the plurality of antennas by, for example, the communication module 2590. can be selected Signals or power may be transmitted or received between the communication module 2590 and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 2597.

According to various embodiments, antenna module 2597 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 2501 and the external electronic device 2504 through the server 2508 connected to the second network 2599. Each of the external electronic devices 2502 or 2504 may be of the same or different type as the electronic device 2501. According to one embodiment, all or part of the operations performed in the electronic device 2501 may be executed in one or more of the external electronic devices 2502, 2504, or 2508. For example, when the electronic device 2501 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 2501 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 2501. The electronic device 2501 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 2501 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 2504 may include an Internet of Things (IoT) device. Server 2508 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, an external electronic device 2504 or a server 2508 may be included in the second network 2599. The electronic device 2501 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

According to one embodiment, the electronic device 110 includes a camera module 1150 that includes a front camera and is configured to output an electrical signal by photographing a subject; A display module 1120 that includes a display panel 1121 or a touch panel 1123 and is configured to display an image on the display panel 1121 or output an electrical signal according to a touch of the touch panel 1123; a communication module 1170 configured to communicate with external electronic devices 120 and 130 based on a network environment; and operation of the camera module 1150, the display module 1120, or the communication module 1170 by electrical connection with the camera module 1150, the display module 1120, or the communication module 1170. It may include at least one processor 1110 configured to control.

According to one embodiment, the at least one processor 1110 may display the avatar of at least one participant among a plurality of participants including the user of the electronic device 110 in a virtual space provided for a conversation function.

According to one embodiment, the at least one processor 1110 may display an utterance indicator indicating an utterance state in response to an utterance by at least one speaker among the plurality of participants.

According to one embodiment, the at least one processor 1110 may control the display module 1120 to change and display the gaze of the avatar in the virtual space based on an utterance by the at least one speaker. there is.

According to one embodiment, the at least one processor 1110 may control the display module 1120 to display the avatar of at least one other participant excluding the user among the plurality of participants in the virtual space. .

According to one embodiment, the at least one processor 1110 is configured to replace the at least one other participant in the virtual space in the direction in which the gaze of the first avatar is directed, centering on the first avatar representing the user. Controls the display module 1120 to display a second avatar, wherein at least one of an angle, an angle of view, or a focus position of a virtual camera is considered for displaying the first avatar or the second avatar in the virtual space. can be changed considering the number of participants.

According to one embodiment, when a plurality of avatars are displayed in the virtual space, the at least one processor 1110 is configured to display the plurality of avatars at different depths in the virtual space (the display module ( 1120) can be controlled.

According to one embodiment, the at least one processor 1110 identifies a change in speaker among the plurality of participants based on data collected from the external electronic devices 120 and 130 through the communication module 1170. Then, a first conversation screen can be configured with a camera angle changed based on the speaker avatar corresponding to the changed speaker, and the display module 1120 can be controlled to display the first conversation screen.

According to one embodiment, the at least one processor 1110 acquires voice characteristics for each participant based on voice data included in the collected data, and combines information on preset reference voice characteristics with the acquired voice characteristics. It may be configured to determine the speaker avatar using .

According to one embodiment, the at least one processor 1110 switches the camera based on an electrical signal provided from the display module 1120 in response to a swipe operation on the touch panel 1123. A second conversation screen can be configured with an angle, and the display module 1120 can be controlled to display the second conversation screen.

According to one embodiment, it includes at least one sensor and includes a sensor module 1160 configured to detect movement of the electronic device 110 in the left/right or up/down directions and output an electrical sensing signal. can do.

According to one embodiment, the at least one processor 1110 configures a third conversation screen with a switched camera angle based on the electrical sensing signal provided from the sensor module 1160, and displays the third conversation screen. The display module 1120 can be controlled to display.

According to one embodiment, the at least one processor 1110 zooms when the speech time by a specific avatar does not exceed a threshold time or in response to a swipe operation on the touch panel 1123 - The display module 1120 is controlled to display a fourth conversation screen provided by a first viewing angle according to the user's input, and the speech time by the specific avatar exceeds the threshold time, or the touch panel 1123 In response to a swipe operation, the display module 1120 may be controlled to display a fifth conversation screen provided by a second angle of view according to zoom-out.

According to one embodiment, the at least one processor 1110 acquires a facial expression or gesture from the user's image captured by the front camera included in the camera module 1150, or from the collected data. It may be configured to acquire the facial expression or gesture of the other participant, and control the display module 1120 to mirror the acquired facial expression or the acquired gesture to the user's avatar or the avatar of the other participant. there is.

According to one embodiment, when the at least one processor 1110 detects that the user has touched the avatar through the touch panel 1123, it calls the external electronic device 130 corresponding to the touched avatar. The communication module 1170 can be controlled to transmit a signal, extract images corresponding to the conversation between the plurality of participants in the virtual space, and create and store a conversation record image using the extracted images.

According to one embodiment, a method of providing a conversation function using an avatar in an electronic device includes displaying an avatar of at least one participant among a plurality of participants including a user of the electronic device in a virtual space provided for the conversation function. ; displaying an utterance indicator indicating an utterance state in response to an utterance by at least one speaker among the plurality of participants; and an operation of changing and displaying the gaze of the avatar in the virtual space based on an utterance by the at least one speaker.

According to one embodiment, the operation of displaying the avatar may include the operation of displaying the avatar of at least one other participant excluding the user among the plurality of participants in the virtual space.

According to one embodiment, the operation of displaying the avatar includes a second avatar representing the at least one other participant in the virtual space centered on the first avatar representing the user and in the direction in which the gaze of the first avatar is directed. An operation of displaying an avatar, wherein at least one of an angle, an angle of view, or a focus position of a virtual camera to be considered for displaying the first avatar or the second avatar in the virtual space or the number of the plurality of participants It can be changed considering the number.

According to one embodiment, the operation of displaying the avatar may include, when a plurality of avatars are displayed in the virtual space, displaying the plurality of avatars to have different depths in the virtual space. there is.

According to one embodiment, the method includes an operation 1013 of identifying that a speaker has changed among the plurality of participants based on data collected from external electronic devices 120 and 130 based on a network environment; An operation of configuring a first conversation screen by switching the camera angle based on the speaker avatar corresponding to the identified speaker (1025); and an operation 1025 of displaying the first conversation screen.

According to one embodiment, the method includes obtaining voice characteristics for each participant based on voice data included in the collected data; and determining the speaker avatar using information on preset reference voice characteristics and the acquired voice characteristics.

According to one embodiment, the method includes configuring a second conversation screen with a camera angle switched in response to a user's swipe operation (1013, 1025); and an operation 1025 of displaying the second conversation screen.

According to one embodiment, the method includes an operation 1013 of configuring a third conversation screen with a switched camera angle based on a left/right or up/down movement of the electronic device 110; and an operation 1027 of displaying the third conversation screen.

According to one embodiment, when the speech time by a specific avatar does not exceed a threshold time, or in response to a user's swipe action, a fourth conversation screen provided by the first angle of view according to zoom-in is output. action(1015, 1027); and an operation of outputting a fifth conversation screen provided by a second angle of view according to zoom-out when the speech time by the specific avatar exceeds the threshold time or in response to a swipe operation of the user (1015) , 1027).

According to one embodiment, the method includes an operation 1017 of acquiring a facial expression or gesture from a user's image captured by a front camera, or acquiring a facial expression or gesture of the other participant from the collected data; And it may include an operation 1029 of mirroring the acquired facial expression or the acquired gesture to the user's avatar or the other participant's avatar.

According to one embodiment, upon detecting that the user has touched the avatar, operations 1021 and 1033 of transmitting a signal to call an external electronic device 130 corresponding to the touched avatar; and extracting images corresponding to the conversation between the plurality of participants taking place in the virtual space, and generating and storing a conversation record image using the extracted images (1035).

According to one embodiment, an electronic device (e.g., the electronic device 110 of FIG. 1) includes a front camera and may include a camera module 1150 configured to output an electrical signal by photographing a subject. The electronic device 110 includes a display panel 1121 or a touch panel 1123, and displays an image on the display panel 1121 or outputs an electrical signal according to a touch of the touch panel 1123. It may include a configured display module 1120. The electronic device 110 may include a communication module 1170 configured to communicate with external electronic devices 120 and 130 based on a network environment. The electronic device 110 is connected to the camera module 1150, the display module 1120, or the communication module 1170 by electrical connection with the camera module 1150, the display module 1120, or the communication module 1170. It may include at least one processor 1110 configured to control the operation of the module 1170.

According to one embodiment, the at least one processor 1110 selects a speaker among attendees who participated in a conversation using an avatar based on data collected from the external electronic devices 120 and 130 through the communication module 1170. It can be identified that has changed. The at least one processor 1110 may configure a first conversation screen with a changed camera angle based on the speaker avatar corresponding to the changed speaker. The at least one processor 1110 may control the display module 1120 to display the first conversation screen on the display panel 1121.

According to one embodiment, the at least one processor 1110 may acquire the voice characteristics of the participants based on the participants' voice data included in the collected data. The at least one processor 1110 may predict a participant having the acquired voice characteristic using information about a preset reference voice characteristic. The at least one processor 1110 may determine the predicted participant's avatar among avatars participating in the conversation as the speaker avatar.

According to one embodiment, the at least one processor 1110 may collect voice data of the attendees through the communication module 1170 when the conversation is established and the conversation is performed by the attendees. The at least one processor 1110 may obtain and register information on standard voice characteristics for each of the attendees using the collected voice data.

According to one embodiment, the at least one processor 1110 switches the camera based on an electrical signal provided from the display module 1120 in response to a swipe operation on the touch panel 1123. You can configure the second conversation screen with an angle. The at least one processor 1110 may control the display module 1120 to display the second conversation screen on the display panel 1121.

According to one embodiment, the electronic device 110 includes at least one sensor, and detects movement of the electronic device 110 in the left/right or up/down directions and outputs an electrical sensing signal. It may include a configured sensor module 1160.

According to one embodiment, the at least one processor 1110 may configure a third conversation screen with a switched camera angle based on the electrical sensing signal provided from the sensor module 1160. The at least one processor 1110 may control the display module 1120 to display the third conversation screen on the display panel 1121.

According to one embodiment, the at least one processor 1110 may configure a fourth conversation screen with a changed view angle based on the collected data. The at least one processor 1110 may control the display module 1120 to display the fourth conversation screen on the display panel 1121.

According to one embodiment, the at least one processor 1110 determines that the speech time by the specific avatar does not exceed the threshold time or in response to a swipe operation on the touch panel 1123. The display module 1120 can be controlled to display the fifth conversation screen provided by the first angle of view according to zoom-in on the display panel 1121.

According to one embodiment, the at least one processor 1110 is configured to zoom in response to a speech time by the specific avatar exceeding the threshold time or a swipe operation on the touch panel 1123. The display module 1120 can be controlled to display a sixth conversation screen provided by a second viewing angle according to -out on the display panel 1121.

According to one embodiment, the at least one processor 1110 may obtain a facial expression or gesture from the user's image captured by the front camera included in the camera module 1150. The at least one processor 1110 controls the display module 1120 to mirror the acquired facial expression or the acquired gesture to the user's avatar included in the conversation screen displayed on the display panel 1121 ( mirroring) is possible.

According to one embodiment, the at least one processor 1110 may obtain facial expressions or gestures of the attendees from the collected data. The at least one processor 1110 controls the display module 1120 to display facial expressions of the obtained attendees or gestures of the obtained attendees included in the conversation screen displayed on the display panel 1121. You can mirror it on your avatar.

According to one embodiment, the at least one processor 1110 may adjust the angle of view in consideration of the number of attendees. The at least one processor 1110 may control the display module 1120 to display a seventh conversation screen provided by the adjusted viewing angle on the display panel 1121.

According to one embodiment, when the at least one processor 1110 detects that the user has touched the avatar of one of the attendees through the touch panel 1123, the at least one processor 1110 detects that the user has touched the avatar of one of the attendees through the communication module 1170. A signal calling the external electronic device 130 corresponding to the avatar may be transmitted.

According to one embodiment, the at least one processor 1110 may extract images corresponding to the conversation based on the collected data. The at least one processor 1110 may generate a conversation record image using the extracted images and store the generated conversation record image.

According to one embodiment, a method of providing a conversation function using an avatar in an electronic device (e.g., the electronic device 110 of FIG. 1) includes data collected from external electronic devices 120 and 130 based on a network environment. Based on this, it may include an operation (610 & 640, 710 & 730, 810 & 880, or 910 & 950) to output the first conversation screen using the avatar. The method may include an operation 1013 of identifying a speaker among attendees based on the collected data. The method may include an operation 1025 of configuring a second conversation screen by switching the camera angle based on the speaker avatar corresponding to the identified speaker. The method may include an operation 1025 of converting the first conversation screen to the second conversation screen and outputting the second conversation screen.

In the method according to one embodiment, the operation of identifying the speaker may include the operation of acquiring the voice characteristics of the participants based on the participants' voice data included in the collected data. The operation of identifying the speaker may include the operation of predicting a participant with the acquired voice characteristic using information about a preset reference voice characteristic. The operation of identifying the speaker may include the operation of determining the avatar of the predicted participant among avatars participating in the conversation as the speaker avatar.

According to one embodiment, the method may include collecting voice data of the attendees when the conversation is opened and the conversation is performed by the attendees. The method may include an operation of obtaining and registering information on standard voice characteristics for each of the attendees using the collected voice data.

According to one embodiment, the method may include operations 1013 and 1025 of configuring a third conversation screen with a camera angle switched in response to a user's swipe operation. The method may include an operation 1025 of converting the conversation screen to the third conversation screen and outputting it.

According to one embodiment, the method may include an operation 1013 of configuring a fourth conversation screen with a switched camera angle based on the left/right or up/down movement of the electronic device 110. You can. The method may include an operation 1025 of converting a conversation screen to the fourth conversation screen and outputting it.

According to one embodiment, the method may include an operation 1015 of configuring a fifth conversation screen with a view angle changed based on the collected data. The method may include an operation 1027 of converting the conversation screen to the fifth conversation screen and outputting it.

According to one embodiment, the method is provided when the speech time by the specific avatar does not exceed the threshold time, or by the first angle of view according to zoom-in in response to the user's swipe operation. 6 May include operations 1015 and 1027 for outputting a conversation screen. The method includes outputting a seventh conversation screen provided by a second angle of view according to zoom-out in response to a speech time by the specific avatar exceeding the threshold time or a user's swipe operation. It may include (1015, 1027).

According to one embodiment, the method may include an operation 1017 of obtaining a facial expression or gesture from an image of the user captured by the front camera. The method may include an operation 1029 of mirroring the acquired facial expression or the acquired gesture to the user's avatar included in the conversation screen.

According to one embodiment, the method may include an operation 1017 of obtaining facial expressions or gestures of the attendees from the collected data. The method may include an operation 1029 of mirroring the acquired facial expressions of the attendees or the acquired gestures of the attendees onto the attendees' avatars included in the conversation screen.

According to one embodiment, the method may include an operation 1019 of adjusting the angle of view in consideration of the number of attendees. The method may include an operation 1031 of outputting an eighth conversation screen provided by the adjusted view angle.

According to one embodiment, the method may include operations 1021 and 1033 of transmitting a signal to call an external electronic device corresponding to the touched avatar when detecting that the user touches a specific avatar on the conversation screen. You can.

According to one embodiment, the method may include operations 1023 and 1035 of extracting images corresponding to the conversation based on the collected data. The method may include an operation 1035 of generating a conversation record image using the extracted images. The method may include an operation 1035 of storing the generated conversation record image.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are software (e.g., computer) including one or more instructions stored in a storage medium (e.g., memory 1180) readable by a machine (e.g., electronic device 110). It may be implemented as a program 2540. For example, a processor (e.g., processor 1110) of a device (e.g., electronic device 110) may execute at least one instruction among one or more instructions stored from a storage medium. and execute it. This enables the device to be operated to perform at least one function according to the called at least one instruction. The one or more instructions are code generated by a compiler or by an interpreter. It may contain executable code. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium is tangible. It simply means that it is a device that does not contain signals (e.g. electromagnetic waves), and this term does not distinguish between cases where data is semi-permanently stored in a storage medium and cases where data is stored temporarily.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Claims (22)

In the electronic device 110,
A camera module 1150 including a front camera and configured to output an electrical signal by photographing a subject;
A display module 1120 that includes a display panel 1121 or a touch panel 1123 and is configured to display an image on the display panel 1121 or output an electrical signal according to a touch of the touch panel 1123;
a communication module 1170 configured to communicate with external electronic devices 120 and 130 based on a network environment; and
The operation of the camera module 1150, the display module 1120, or the communication module 1170 is controlled by electrical connection with the camera module 1150, the display module 1120, or the communication module 1170. It includes at least one processor 1110 configured to control,
The at least one processor 1110,
Displaying the avatar of at least one participant among a plurality of participants including the user of the electronic device 110 in a virtual space provided for a conversation function,
Displaying an utterance indicator indicating an utterance state in response to an utterance by at least one speaker among the plurality of participants,
An electronic device that controls the display module 1120 to change and display the gaze of the avatar in the virtual space based on an utterance by the at least one speaker. According to paragraph 1,
The at least one processor 1110,
An electronic device that controls the display module 1120 to display the avatar of at least one other participant, excluding the user, among the plurality of participants in the virtual space. According to paragraph 2,
The at least one processor 1110,
Controlling the display module 1120 to display a second avatar representing the at least one other participant in the virtual space centered on the first avatar representing the user and in the direction in which the first avatar's gaze is directed,
An electronic device that changes at least one of an angle, an angle of view, or a focus position of a virtual camera to be considered for displaying the first avatar or the second avatar in the virtual space in consideration of the number of participants. . According to paragraph 1,
The at least one processor 1110,
When a plurality of avatars are displayed in the virtual space, an electronic device that controls the display module 1120 to display the plurality of avatars at different depths in the virtual space. According to paragraph 1,
The at least one processor 1110,
Identifying a change in speaker among the plurality of participants based on data collected from the external electronic devices 120 and 130 through the communication module 1170,
Constructing a first conversation screen with a camera angle switched based on the speaker avatar corresponding to the changed speaker,
An electronic device 110 that controls the display module 1120 to display the first conversation screen. According to clause 5,
The at least one processor 1110,
Obtain voice characteristics for each participant based on the voice data included in the collected data,
The electronic device 110 is configured to determine the speaker avatar using information about preset reference voice characteristics and the acquired voice characteristic. According to claim 5 or 6,
The at least one processor 1110,
Constructing a second conversation screen with a switched camera angle based on an electrical signal provided from the display module 1120 in response to a swipe operation on the touch panel 1123,
An electronic device 110 that controls the display module 1120 to display the second conversation screen. According to claim 5 or 6,
It includes at least one sensor, and includes at least a sensor module 1160 configured to detect movement of the electronic device 110 in the left/right or up/down directions and output an electrical sensing signal,
The at least one processor 1110,
Constructing a third conversation screen with a camera angle switched based on the electrical sensing signal provided from the sensor module 1160,
An electronic device 110 that controls the display module 1120 to display the third conversation screen. According to clause 5,
The at least one processor 1110,
Displaying a fourth conversation screen provided by a first angle of view according to zoom-in when the speech time by a specific avatar does not exceed a threshold time or in response to a swipe operation on the touch panel 1123 Controlling the display module 1120 so as to
When the speech time by the specific avatar exceeds the threshold time, or in response to a swipe operation on the touch panel 1123, displaying a fifth conversation screen provided by a second angle of view according to zoom-out An electronic device 110 that controls the display module 1120 to do so. According to clause 5,
The at least one processor 1110,
Obtaining facial expressions or gestures from the user's image captured by the front camera included in the camera module 1150, or acquiring facial expressions or gestures of the other participant from the collected data,
The electronic device 110 is configured to control the display module 1120 to mirror the acquired facial expression or the acquired gesture to the user's avatar or the other participant's avatar. According to paragraph 1,
The at least one processor 1110,
When detecting that the user has touched the avatar through the touch panel 1123, the communication module 1170 is controlled to transmit a signal to call an external electronic device 130 corresponding to the touched avatar,
Extracting images corresponding to conversations between the plurality of participants taking place in the virtual space,
An electronic device 110 that generates and stores a conversation record image using the extracted images. In a method of providing a conversation function using an avatar in an electronic device,
Displaying an avatar of at least one participant among a plurality of participants including a user of the electronic device in a virtual space provided for the conversation function;
displaying an utterance indicator indicating an utterance state in response to an utterance by at least one speaker among the plurality of participants; and
A method comprising changing and displaying the gaze of the avatar in the virtual space based on an utterance by the at least one speaker. According to clause 12,
The operation of displaying the avatar is,
A method comprising displaying in the virtual space the avatar of at least one other participant, excluding the user, among the plurality of participants. According to clause 13,
The operation of displaying the avatar is,
Including the operation of displaying a second avatar to replace the at least one other participant in the virtual space centered on the first avatar representing the user and in the direction in which the first avatar's gaze is directed,
A method of changing at least one of an angle, an angle of view, or a focus position of a virtual camera to be considered for displaying the first avatar or the second avatar in the virtual space in consideration of the number of participants. According to clause 12,
The operation of displaying the avatar is,
When a plurality of avatars are displayed in the virtual space, a method comprising displaying the plurality of avatars to have different depths in the virtual space. According to clause 12,
An operation 1013 of identifying a change in speaker among the plurality of participants based on data collected from external electronic devices 120 and 130 based on a network environment (1013);
An operation of configuring a first conversation screen by switching the camera angle based on the speaker avatar corresponding to the identified speaker (1025); and
A method comprising an operation (1025) of displaying the first conversation screen. According to clause 16,
Obtaining voice characteristics for each participant based on voice data included in the collected data; and
A method comprising determining the speaker avatar using information about preset reference voice characteristics and the acquired voice characteristic. According to claim 16 or 17,
An operation (1013, 1025) of configuring a second conversation screen with a switched camera angle in response to a user's swipe operation; and
A method comprising an operation (1025) of displaying the second conversation screen. According to claim 16 or 17,
An operation 1013 of configuring a third conversation screen with a switched camera angle based on the left/right or up/down movement of the electronic device 110; and
A method comprising an operation (1027) of displaying the third conversation screen. According to clause 16,
An operation of outputting a fourth conversation screen provided by the first angle of view according to zoom-in when the speech time by a specific avatar does not exceed the threshold time or in response to the user's swipe action (1015, 1027) ; and
An operation (1015) of outputting a fifth conversation screen provided by a second angle of view according to zoom-out in response to a speech time by the specific avatar exceeding the threshold time or a swipe operation of the user. 1027), including a method. According to clause 16,
An operation 1017 of acquiring a facial expression or gesture from a user's image captured by a front camera, or acquiring a facial expression or gesture of the other participant from the collected data; and
A method comprising an operation 1029 of mirroring the acquired facial expression or the acquired gesture to the user's avatar or the other participant's avatar. According to clause 12,
When detecting that the user has touched the avatar, transmitting a signal to call an external electronic device 130 corresponding to the touched avatar (1021, 1033); and
A method comprising the operation (1035) of extracting images corresponding to conversations between the plurality of participants taking place in the virtual space, and generating and storing conversation record images using the extracted images.

Images