KR20200092207A - Electronic device and method for providing graphic object corresponding to emotion information thereof - Google Patents

Abstract

According to various embodiments of the present invention, an electronic device comprises a display, a microphone, a processor operatively connected to the display and the microphone, and a memory operatively connected to the processor. The memory may store instructions such that the processor displays an image on the display, recognizes at least one object from the displayed image, extracts at least one feature area for the recognized at least one object, determines first emotion information based on received one or more voice data if the one or more voice data are received via the microphone, generates at least one graphic object associated with the one or more voice data based on the first emotion information, determines an area where the generated at least one graphic object is to be synthesized in the at least one feature area based on the speech content of the one or more voice data, and synthesizes the generated at least one graphic object at a position corresponding to the determined area when executed. Various embodiments other than the various embodiments disclosed in the present invention are possible.

Description

Electronic device and method of providing graphic object corresponding to emotion information using the same {ELECTRONIC DEVICE AND METHOD FOR PROVIDING GRAPHIC OBJECT CORRESPONDING TO EMOTION INFORMATION THEREOF}

Various embodiments of the present invention relate to a method of providing a graphic object corresponding to emotion information.

Recently, online communities such as social network services (SNS) and messenger services have been activated, so that users can share their daily lives with others. For example, a user writes his or her daily life or thoughts in writing, shares it with the other person, emojis with the text, and a predefined emoji based on the facial feature area extracted from the user's image (eg, hairstyle, glasses, hat) , Stickers) to express their feelings directly or indirectly to the other party.

The electronic device provides a user with a predefined emoticon or emoji, but is limited in expressing the user's emotions because the predefined types of emoticons or emoji are limited. In addition, in an AR (augmented reality) environment, the user cannot take a picture of his face, and accordingly, cannot transmit his feelings to the other party.

The electronic device according to various embodiments of the present disclosure may determine emotion information of an object based on a feature region of an object included in an image, and provide a graphic object corresponding to the determined emotion information.

The electronic device according to various embodiments of the present disclosure may determine emotion information based on input voice data, and provide graphic objects related to the voice data corresponding to the emotion information by mapping the recognized object in the image.

The electronic device according to various embodiments of the present disclosure may apply one's emotion to another object (eg, an animal or an object) obtained from the camera in an AR environment in which the user's own face cannot be captured.

According to various embodiments of the present invention, an electronic device includes a display, a microphone, a processor operatively connected to the display and the microphone, and a memory operatively connected to the processor, wherein the memory, upon execution , The processor displays an image on the display, recognizes at least one object from the displayed image, extracts at least one feature region for the recognized at least one object, and at least one through the microphone When voice data is received, first emotion information is determined based on the received at least one voice data, and at least one graphic object related to the at least one voice data is generated based on the first emotion information, A region to synthesize the generated at least one graphic object among the at least one feature region is determined based on the content of speech of the at least one voice data, and the generated at least one location at a position corresponding to the determined region. You can store instructions that let you compose a graphic object.

According to various embodiments of the present invention, an electronic device includes a display, a processor operatively connected to the display, and a memory operatively connected to the processor, wherein the memory, when executed, includes the processor, Displaying an image on the display, recognizing at least one object from the displayed image, and determining emotion information based on at least one of data related to the image, context information, or information related to a user of the electronic device, When an area in the image to synthesize at least one graphic object corresponding to the determined emotion information is determined, the display displays one or more graphic objects corresponding to the determined emotion information to be synthesized in the area based on characteristics of the determined area. Instructions to display and synthesize the at least one selected graphic object at a position corresponding to the determined area when a user input for selecting at least one of the displayed one or more graphic objects is detected. ).

According to various embodiments of the present invention, a method of providing a graphic object corresponding to emotion information of an electronic device includes displaying an image on a display, recognizing at least one object from the displayed image, and recognizing the recognized image. Extracting at least one feature region for at least one object, determining at least one voice information based on the received at least one voice data when at least one voice data is received through the microphone, the Generating at least one graphic object related to the at least one voice data based on first emotion information, and the generated at least one of the at least one feature region based on the utterance content of the at least one voice data The method may include determining an area to synthesize the graphic object, and synthesizing the generated at least one graphic object at a position corresponding to the determined area.

According to various embodiments of the present invention, a method of providing a graphic object corresponding to emotion information of an electronic device includes: displaying an image on a display, recognizing at least one object from the displayed image, and Determining emotion information based on at least one of related data, context information, or information related to a user of the electronic device, and determining an area in the image to synthesize at least one graphic object corresponding to the determined emotion information , Displaying one or more graphic objects corresponding to the determined emotion information to be synthesized in the area on the display based on characteristics of the determined area, and selecting at least one graphic object from the displayed one or more graphic objects When a user input is sensed, it may include an operation of synthesizing the selected at least one graphic object at a position corresponding to the determined area.

The electronic device according to various embodiments of the present disclosure may determine emotion information for an image based on data related to the image or input voice data, and synthesize a graphic object corresponding to the determined emotion information into the image. The user can communicate with the other party by using the image in which the graphic object corresponding to the emotion information is synthesized, so that the user's emotion can be more abundantly transmitted to the other party.

In an electronic device according to various embodiments of the present disclosure, in an AR environment in which a user's own face cannot be photographed, the emotions are reinterpreted according to a characteristic area of a person in an object (or animal) obtained from the camera. It can provide a fun factor to the user.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram illustrating an electronic device according to various embodiments of the present disclosure.
3 is a flowchart illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.
4 is a flowchart illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.
5 is a diagram for describing a method of determining emotion information by extracting a feature region for at least one object recognized from an image according to various embodiments.
6 is a diagram for describing a method of generating at least one graphic object based on determined emotion information according to various embodiments.
7 is a diagram for describing a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
8 is a diagram for explaining a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
9A and 9B are diagrams illustrating a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
10 is a diagram for describing a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
11 is a diagram illustrating signal flow between electronic devices that provide a graphic object corresponding to emotion information according to various embodiments.
12 is a diagram illustrating a signal flow between electronic devices providing corresponding graphic objects corresponding to emotion information according to various embodiments.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments.

Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through the first network 198 (eg, a short-range wireless communication network), or the second network 199. It may communicate with the electronic device 104 or the server 108 through (eg, a remote wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, a sensor module ( 176), interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ). In some embodiments, at least one of the components (for example, the display device 160 or the camera module 180) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented in one integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, the program 140) to execute at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to one embodiment, as at least part of data processing or computation, the processor 120 may receive instructions or data received from other components (eg, the sensor module 176 or the communication module 190) in the volatile memory 132. Loaded into, process instructions or data stored in volatile memory 132, and store result data in non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and an auxiliary processor 123 (eg, a graphics processing unit, an image signal processor) that can be operated independently or together. , Sensor hub processor, or communication processor). Additionally or alternatively, the coprocessor 123 may be set to use less power than the main processor 121, or to be specialized for a specified function. The coprocessor 123 may be implemented separately from the main processor 121 or as a part thereof.

The coprocessor 123 may replace, for example, the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 may be active (eg, execute an application) ) With the main processor 121 while in the state, at least one component of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It can control at least some of the functions or states associated with. According to one embodiment, the coprocessor 123 (eg, image signal processor or communication processor) may be implemented as part of other functionally relevant components (eg, camera module 180 or communication module 190). have.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176). The data may include, for example, input data or output data for software (eg, the program 140) and instructions related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.

The input device 150 may receive commands or data to be used for components (eg, the processor 120) of the electronic device 101 from outside (eg, a user) of the electronic device 101. The input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, a stylus pen).

The audio output device 155 may output an audio signal to the outside of the electronic device 101. The audio output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from, or as part of, the speaker.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry configured to sense a touch, or a sensor circuit (eg, a pressure sensor) configured to measure the strength of the force generated by the touch. have.

The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 acquires sound through the input device 150, or an external electronic device (eg, directly or wirelessly connected to the sound output device 155 or the electronic device 101) Sound may be output through the electronic device 102 (eg, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biological sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more designated protocols that can be used for the electronic device 101 to directly or wirelessly connect to an external electronic device (eg, the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or movement) or electrical stimuli that the user can perceive through tactile or motor sensations. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and videos. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishing and performing communication through the established communication channel. The communication module 190 operates independently of the processor 120 (eg, an application processor) and may include one or more communication processors supporting direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may include a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : Local area network (LAN) communication module, or power line communication module. Corresponding communication module among these communication modules includes a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network 199 (eg, a cellular network, the Internet, Or, it may communicate with an external electronic device through a computer network (eg, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into a single component (eg, a single chip), or may be implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses a subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 in a communication network such as the first network 198 or the second network 199. The electronic device 101 can be identified and authenticated.

The antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive it from the outside. According to an embodiment, the antenna module 197 may include a single antenna including a conductor formed on a substrate (eg, a PCB) or a radiator made of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network 198 or the second network 199, is transmitted from the plurality of antennas by, for example, the communication module 190. Can be selected. The signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as part of the antenna module 197.

At least some of the components are connected to each other through a communication method between peripheral devices (for example, a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)). Ex: command or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be the same or a different type of device from the electronic device 101. According to an embodiment, all or some of the operations performed on the electronic device 101 may be performed on one or more external electronic devices of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead executes the function or service itself. In addition or in addition, one or more external electronic devices may be requested to perform at least a portion of the function or the service. The one or more external electronic devices receiving the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and deliver the result of the execution to the electronic device 101. The electronic device 101 may process the result, as it is or additionally, and provide it as at least part of a response to the request. To this end, cloud computing, distributed computing, or client-server computing technology can be used, for example.

2 is a block diagram 200 illustrating an electronic device according to various embodiments.

Referring to FIG. 2, the electronic device 201 (eg, the electronic device 101 of FIG. 1) includes a wireless communication circuit 210 (eg, the communication module 190 of FIG. 1 ), a camera 220 (eg: Camera module 180 of FIG. 1, memory 230 (eg, memory 130 of FIG. 1), sensor unit 240 (eg, sensor module 176 of FIG. 1), audio processing circuit 250 (Eg, the audio module 170 of FIG. 1), a touch screen display 260 (eg, the display device 160 of FIG. 1), and a processor 270 (eg, the processor 120 of FIG. 1). can do.

According to one embodiment, the wireless communication circuit 210 (eg, the communication module 190 of FIG. 1) includes an electronic device 201 (eg, the electronic device 101 of FIG. 1) and an external electronic device (eg: Communication between the electronic device 102, the electronic device 104 of FIG. 1, or a server (eg, the server 108 of FIG. 1) may be connected.

According to one embodiment, the camera 220 (eg, the camera module 180 of FIG. 1) transmits the collected image as a preview image to the display 261 so that the user can view the image projected through the camera 220. Can be confirmed. The camera 220 may generate an image data by photographing the collected image at the time when the input requesting the photographing occurs in response to an input requesting photographing. In one embodiment, one or more cameras 220 may be provided.

According to one embodiment, the memory 230 (eg, the memory 130 of FIG. 1) is a program for generating and synthesizing a graphic object corresponding to emotion information into an image and a program according to embodiments of the present invention. A graphic object composite image may be stored. The memory 230 may store a reference value for determining emotion information based on information related to a feature region extracted from an object, data in an image, context information, and voice data. The memory 230 may store at least one graphic object corresponding to emotion information. The memory 230 may store attribute values corresponding to emotion information to be applied to text for voice data. The memory 230 may store an image obtained by combining at least one graphic corresponding to emotion information.

According to an embodiment, the sensor unit 240 (eg, the sensor module 176 of FIG. 1) detects an external environmental state (eg, a user state) of the electronic device 201 and generates electricity corresponding to the sensed state. You can generate signal or data values. For example, the sensor unit 240 may include a gesture sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

According to an embodiment, the audio processing circuit 250 (eg, the audio module 170 of FIG. 1) is not illustrated, but may include a speaker and a microphone. The audio processing circuit 250 may further include an earphone and a microphone. In one embodiment, the microphone may receive sound (eg, voice data), convert it into an electrical signal, and provide it to the processor 270.

According to one embodiment, the touch screen display 260 (eg, the display device 160 of FIG. 1) may be configured as an integral type including the display 261 and the touch panel 263.

According to one embodiment, the touch screen display 260 may display an image under the control of the processor 270. The touch screen display 260 may display at least one graphic object corresponding to emotion information generated under the control of the processor 270. The touch screen display 260 may synthesize at least one object generated in a specific area in the image under the control of the processor 270 and display an image in which the at least one object is synthesized.

According to one embodiment, the processor 270 (eg, the processor 120 of FIG. 1) controls the overall operation of the electronic device 201 and the signal flow between the internal components of the electronic device 201, and processes data. And control the supply of power from the battery to the components.

According to one embodiment, the processor 270 may display an image on the display 261. For example, the displayed image may include at least one image obtained from at least one camera 220, or an image stored in the memory 230. The processor 270 can recognize at least one object from the displayed image. have. For example, the at least one object may include at least one of a person, an animal, or an object. The processor 270 may extract at least one feature region (eg, a facial feature region including an eye region, a nose region, and/or a mouth region) for the recognized at least one object.

In one embodiment, the processor 270 may determine emotion information based on at least one of the extracted at least one feature area, image-related data, or context information. For example, the data related to the image includes the background of the image, the object placed in the foreground, the type of object (e.g. gun, knife, pointed object), the color of the image (e.g. saturation, brightness), within the image The included audio and/or text included in the video may be included, and the context information may include a current time, a current location, and/or an event schedule.

In one embodiment, when at least one voice data is received from at least one user through a microphone (not shown), in one embodiment, the processor 270 includes a speech content for the received at least one voice data, each of the at least one voice data It may analyze at least one of the included user's voice tone, pitch, frequency, or sound pressure. The processor 270 may determine emotion information for each of the at least one voice data based on the analyzed information. The processor 270 may generate at least one graphic object to be synthesized in an image based on the determined emotion information for each of the at least one voice data. For example, the at least one graphic object may include a sticker corresponding to the determined emotion information, an emoji, and/or text applied with attributes related to the at least one voice data.

In one embodiment, the processor 270 determines an area to synthesize at least one graphic object generated among the at least one feature area based on the content of speech of at least one voice data, and generates at least a location generated in the determined image. One graphic object can be composited. The processor 270 may store an image in which at least one graphic object is synthesized in the memory 230.

3 is a flowchart 300 illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.

Referring to FIG. 3, the processor (eg, the processor 270 of FIG. 2) may display an image on the display (eg, the display 261 of FIG. 2) in operation 301. For example, the displayed image may include at least one image obtained from at least one camera (eg, the camera 220 of FIG. 2 ), or an image stored in a memory (eg, the memory 230 of FIG. 2 ). .

In one embodiment, in operation 303, the processor may recognize at least one object from the displayed image. For example, the at least one object may include at least one of a person, an animal, or an object. In one embodiment, the at least one object may be automatically recognized, or may be recognized by a user input selecting at least one object in an image.

In one embodiment, in operation 305, the processor may extract at least one feature region for the recognized at least one object. For example, the at least one feature region is, for example, a facial feature region, and may include an eye region, a nose region, and/or a mouth region.

In one embodiment, the at least one feature region may be automatically extracted, or may be extracted by a user input selecting at least one feature region. Control points (eg, markers) may be generated on the contours of the extracted at least one feature area, for example, the contours of the left and right eye areas, the contours of the nose area, and/or the contours of the mouth area. In another embodiment, the processor may sense a user input adjusting a control point generated in the contour of the extracted at least one feature area, and adjust the extracted at least one feature area based on the detected user input. At least one feature region for at least one object may be accurately extracted through a user input adjusting a control point.

In one embodiment, the processor may further perform an operation of tracking changes in at least one feature region for at least one object based on a difference image between image frames. For example, a change in at least one feature area, for example, eye movement or mouth movement may be used to determine first emotion information of operation 307 described later.

In one embodiment, the processor determines the first emotion information based on at least one of the extracted at least one feature region, the image-related data, or context information in operation 307. For example, the first emotion information may include at least one of happiness, anger, anxiety, sadness, boredom, disgust, or daily life.

In one embodiment, the processor may determine the first emotion information based on the extracted at least one feature region, for example, the distance between the left and right eyes and/or the shape of the mouth. The present invention is not limited thereto, and the processor may determine the first emotion information based on the movement of at least one feature region, for example, the movement of the eyes, movement of the lips, and/or movement of the face.

In one embodiment, data related to an image includes an image background, an object disposed in the foreground, an object type (eg, gun, knife, pointed object), and an image color (eg, saturation, brightness). , Audio included in an image, and/or text included in an image. The processor may determine first emotion information based on data related to the image.

In one embodiment, the context information may include the current time, current location, and/or event schedule. The processor may determine the first emotion information based on the context information. For example, the first emotion information may be determined by analyzing an event according to the current time. When the event over time is birthday or Christmas, the first emotion information may be determined as positive emotion information (for example, happiness). When the event over time is Halloween, the first emotion information is negative emotion information (for example: Fear). Alternatively, based on the current location, for example, in the case of a play garden, the first emotion information may be determined as positive emotion information (eg, happiness). If the current location is a ghost house, the first emotion information may be negative emotion information (eg: Fear).

In another embodiment, the priority may be set in at least one feature area, data related to an image, or context information. The processor may determine the first emotion information based on at least one feature area, image-related data, or context information having a higher priority.

In an embodiment, in operation 309, the processor may receive at least one voice data through a microphone (eg, the audio processing circuit 250 of FIG. 2). For example, at least one voice data may be received from at least one user. In operation 311, the processor may analyze the received at least one voice data, and based on this, determine second emotion information for each of the at least one voice data. The second emotion information may include at least one of happiness, anger, anxiety, sadness, boredom, disgust, or daily life. For example, when at least one voice data is received, the processor may select at least one of speech content for each of the received at least one voice data, a user's voice tone, pitch, frequency, or sound pressure included in each of the at least one voice data. Can be analyzed.

In one embodiment, the processor may generate at least one graphic object to be synthesized in the image based on at least one of the first emotion information or the second emotion information in operation 313. For example, at least one graphic object is applied with attributes related to at least one voice data generated based on stickers, emojis, and/or second emotion information corresponding to at least one of the first emotion information or the second emotion information. Can contain text. For example, the text related to the at least one voice data generated based on the second emotion information may include text to which at least one of attributes corresponding to the second emotion information, such as font size, style, color, thickness, or motion, is applied. Can.

In another embodiment, when priority is set in the first emotion information and the second emotion information, the processor may synthesize at least one of the first emotion information and the second emotion information in the image based on the emotion information having the highest priority. You can create a graphic object.

In one embodiment, when the text to which the attribute corresponding to the second emotion information is applied is generated, the processor may display the text to which the attribute is applied on the display. In one embodiment, when there are multiple texts to which an attribute has been applied, the processor may detect a user input selecting one of the plurality of texts. The processor may synthesize text selected by a user input into an image.

In one embodiment, the text to which the attribute is applied may be customized by the user. At least one graphic object corresponding to emotion information may be generated through customizing the text to which the attribute is applied.

In one embodiment, in operation 315, the processor may determine an area to synthesize at least one graphic object generated among the at least one feature area based on the content of speech of the at least one voice data. For example, if the speech content of the at least one voice data is content related to the mouth (eg, “delicious”), the processor determines, for example, the mouth region as the region to synthesize the at least one graphic object generated from among the at least one feature region. Can. For another example, if the speech content of the at least one voice data is content related to the mouth (eg “delicious”), and content related to the nose (eg “smells good”), the processor may, for example, select one of the at least one feature area, for example. , The mouth region and the nose region may be determined as regions to synthesize at least one generated graphic object.

However, the present invention is not limited thereto, and an area to synthesize the at least one graphic object may be determined based on a user input for selecting a position in the image to place the at least one graphic object.

In one embodiment, in operation 317, the processor may synthesize at least one graphic object generated at a position in the determined image. In operation 319, the processor may store an image in which at least one graphic object is synthesized in a memory.

After generating at least one graphic object in operation 313 in FIG. 3 according to an embodiment, operation 315 is described, but is not limited thereto. After generating at least one graphic object in operation 313, the processor may further receive voice data from the user.

In one embodiment, it has been described as generating at least one graphic object to be synthesized in an image based on at least one of the first emotion information or the second emotion information in operation 313, but is not limited thereto. For example, the processor may generate at least one graphic object based on the first emotion information from which the first emotion information of operation 307 is determined, and provide a list for it. After determining the second emotion information in operation 311, the processor may generate at least one graphic object based on the second emotion information and provide a list for it. The processor may determine the third emotion information based on the first emotion information and the second emotion information. The processor may generate at least one graphic object related to the third emotion information based on the third emotion information, and provide a list for it. The user can select a graphic object to be synthesized in the image from a list corresponding to each emotion information. As the third emotion information is determined by fusion of the first emotion information and the second emotion information, and a graphic object is provided, an optimal graphic object for at least one object recognized in the image and the image may be obtained.

4 is a flowchart 400 illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.

Referring to FIG. 4, a processor (for example, the processor 270 of FIG. 2) may display an image in operation 401. In operation 403, the emotion information may be determined based on at least one of data related to an image, context information, or information related to a user. For example, the information related to the image may include audio information included in the image and/or motion information of at least one object included in the image. The context information may include the current time, current location, and/or event schedule. The information related to the user may include at least one biometric information, at least one gesture information, and/or at least one voice data.

In one embodiment, in operation 405, the processor may determine an area in the image to synthesize the at least one graphic object. For example, an area in which at least one graphic object is to be placed may be determined based on a user input for selecting an area to synthesize at least one graphic object from an image. In operation 407, the processor may provide one or more graphic objects corresponding to emotion information to be synthesized based on the determined characteristics of the region. For example, one or more provided graphic objects may be provided differently based on the determined region in the image. If the determined region in the image is an eye region, eye images corresponding to emotion information may be provided. If the area in the determined image is an area in which the animation character object is displayed, corresponding character images in which emotion information is reflected in the animation character object may be provided. If the area in the determined image is a region displaying a background image, the color, brightness, and/or transparency of the background image excluding the foreground may be determined based on emotion information and provided.

In one embodiment, in operation 409, the processor may detect an input for selecting at least one graphic object to be synthesized in an image from among the provided one or more graphic objects. In operation 411, the at least one graphic object selected at the determined location in the image may be synthesized, and in operation 413, the image obtained by combining the at least one graphic object may be stored in the memory.

In another embodiment, although not shown, the processor may determine emotion information based on at least one object selected by a user input. The processor may generate at least one graphic object corresponding to the determined emotion information, and synthesize the generated at least one graphic object in an image.

In another embodiment, although not shown, the processor may determine a plurality of emotion information based on the plurality of objects when there are a plurality of objects selected by the user input. The processor may provide at least one graphic object to be synthesized in an image generated based on each of the determined emotion information. For example, the processor may provide at least one graphic object generated based on the first emotion information and at least one graphic object generated based on the second emotion information. The processor may synthesize the selected graphic object in the image in response to an input of selecting one of the at least one graphic object generated based on each emotion information provided.

In another embodiment, although not shown, the processor may segment an image into a specific number of regions, and recognize one or more objects included in the divided region. The processor may determine emotion information based on each of the recognized one or more objects. The processor may provide at least one graphic object to be synthesized in the divided region based on each of the determined emotion information. The processor may synthesize the selected graphic object in the corresponding area in response to detection of an input for selecting one of the at least one graphic object generated based on each emotion information for each area provided.

In another embodiment, although not shown, the processor may determine the emotion information based on each of the one or more objects recognized in the image, and then receive a plurality of voice data. When a plurality of voice data is received, the processor may segment the received plurality of voice data. For example, the processor may divide the received plurality of voice data based on the speech rate, sound tone, pitch, frequency, and/or sound pressure for each of the received plurality of voice data. The processor may determine emotion information for each of the plurality of voice data based on at least one of speech content, sound tone, pitch, frequency, or sound pressure of each of the received plurality of voice data. The processor may provide at least one graphic object based on the emotion information for each of the determined voice data. The processor may generate at least one graphic object corresponding to emotion information for each of the determined voice data. The processor may determine at least one object to be synthesized based on emotion information of each of the one or more objects, and synthesize at least one object determined in a corresponding feature area of each of the one or more objects. In another embodiment, when there are a plurality of generated at least one graphic object, a user input for selecting at least one graphic object among the plurality of graphic objects may be sensed. The processor may synthesize the selected at least one graphic object in the feature area of each of the recognized one or more objects in response to a user input.

5 is a diagram 500 illustrating a method of determining emotion information by extracting a feature region for at least one object recognized from an image according to various embodiments.

Referring to FIG. 5, at least one feature region (eg, at least one of a human 550, an object 560, or an animal 570) that is a reference for determining emotion information (at least one feature region) For example: characteristics of the eye area, nose area, and/or mouth area may be stored in a memory (eg, the memory 230 of FIG. 2). For example, the characteristics of the at least one feature region may include at least one of the shape of the eye (eg, the degree of sagging of the eye, the focus of the eye, the size of the eye, and the wrinkles of the eye) or the shape of the mouth. The processor (eg, the processor 270 of FIG. 2) may determine emotion information for each of the at least one object based on the characteristics of the extracted at least one feature region. For example, emotional information may include sadness 510, angry 520, surprise 530, and/or happiness 540. Although not limited thereto, although not shown, the emotion information may include anxiety, boredom, and/or hate.

6 is a diagram 600 illustrating a method of generating at least one graphic object based on determined emotion information according to various embodiments.

Referring to FIG. 6, a graphic object generated based on emotion information and attributes of the graphic object may be stored in a memory (eg, the memory 230 of FIG. 2 ). For example, the processor (for example, the processor 270 of FIG. 2) may include a graphic object based on emotion information stored in the memory and emotion information determined based on the attributes of the graphic object, such as happiness 610, surprise ( 620), at least one of a font style 650, a color 660, a motion 670, or a sticker 680 corresponding to each emotion information according to the angry 630 or sad 640 You can provide one.

7 is a diagram 700 illustrating a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.

Referring to FIG. 7, the processor recognizes the face 713 of the object 711 included in the image, as shown in reference numerals <710> and <720>, and at least one feature from the recognized face 713 Regions, for example, the eye region 721 and the mouth region 723 can be extracted. The processor may determine emotion information on the recognized object 711 based on the extracted eye region 721 and the mouth region 723. For example, based on the shape of the extracted eye region 721 and the mouth region 723, the processor may determine emotion information, for example, as “angry”.

In one embodiment, the processor may analyze the background 731 of the image (eg, an area excluding the object 711 recognized in the image) as illustrated by reference numeral <730>. The processor may analyze the background 731 of the image and determine an atmosphere for the image based on this. For example, an atmosphere for an image may be determined based on the color (or saturation and brightness) of the background 731. However, the present invention is not limited to this, and the processor includes the types of objects included in the image (for example, objects such as weapons and skeletons, objects such as animals and dolls), and the movement of the objects (for example, fast movement and slow movement), and the frequency of the image. The mood for the image may be determined based on the components (eg, edge, high-frequency component of the boundary line, low-frequency component), and/or text included in the image. For example, if an object such as a weapon or a skeleton is recognized, or if the movement of the object exceeds a specified speed, or if the frequency component of the image is a high frequency component, or if the text contained in the image is text representing negative emotions, the processor sends emotion information. You can decide to be “horror” or “anger”. If an object, such as an animal or doll, is recognized, or if the movement of the object is below a specified speed, or if the frequency component of the image is a low-frequency component, or if the text contained in the image is text representing positive emotions, the processor will send the emotion information to “Happiness. ”, “Pleasure”.

The atmosphere determined based on the image in FIG. 7 according to an embodiment will be described on the assumption that it is a creepy and scary atmosphere.

In one embodiment, the processor includes an object 711 recognized from an image, at least one feature region 721, 723 for the object 711, determined emotional information (eg angry), and mood (eg spooky and scary) The atmosphere) may be stored in a memory (eg, the memory 230 of FIG. 2).

In one embodiment, the processor is based on the extracted eye 721 and the shape of the mouth 723, at least one graphic object based on the emotion information (eg, angry) determined and the determined atmosphere (eg, spooky and scary atmosphere) Can generate

In one embodiment, the processor may receive voice data such as “watch out” 735 through the microphone. The processor may analyze the received voice data, such as the content of the “watch out” 735, the characteristics of the user voice included in the voice data (eg, tone, pitch, frequency, and/or sound pressure).

In one embodiment, the processor determines emotion information for the input voice data based on the content of the analyzed voice data and characteristics of the user's voice, as shown in reference numeral <740>, and based on this, the at least one graphic An object 741 may be generated. Based on the analysis result of the voice data, emotion information may be determined, for example, as “angry”.

In one embodiment, the processor may display at least one graphic object 751 to which an attribute corresponding to the generated emotion information is applied as shown in reference numeral <750>. The attribute may include at least one of font size, style, color, thickness, or motion. For example, the at least one graphic object to which the attribute is applied may include voice data, for example, text generated based on “watch out” 735, stickers generated based on emotion information and mood.

In one embodiment, the processor may display a confirmation button 753 with at least one graphic object 751 to which an attribute corresponding to emotion information is applied. When a user input for selecting the confirmation button 753 is detected, the processor is based on at least one graphic object to which an attribute is applied, for example, emotion information (eg, anger) for voice data as shown in reference numeral <760>. Generated text 751 (e.g., font with sharp angled border and text with red color applied) and sticker 761 created based on emotion information (e.g. angry) and mood (e.g. spooky and scary atmosphere) (e.g. You can compose a sticker of angry eyes) in the video. In an embodiment, each of the text 751 and the sticker 761 may be synthesized by being disposed in a corresponding feature region, for example, the mouth region 723 and the eye region 721 among at least one extracted feature region.

In the above-described embodiment, as shown in reference numeral <750>, in response to a user input selecting the confirmation button 753, an image in which at least one graphic object to which the attribute of reference numeral <760> is applied is displayed is displayed. However, it is not limited to this. For example, after displaying at least one graphic object 751 to which the attribute corresponding to the generated emotion information of reference number <750> is applied, if the specified time (for example, 3 seconds) is exceeded, the processor displays reference number <760>. An image obtained by combining at least one graphic object to which an attribute is applied may be displayed.

In one embodiment, although not shown, the processor may display a button for customizing the graphic object 751 together with at least one graphic object 751 to which attributes corresponding to emotion information are applied. At least one graphic object corresponding to emotion information intended by the user may be accurately generated through a user input of selecting a button for customizing.

8 is a diagram 800 for describing a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.

Referring to FIG. 8, the processor recognizes the face of the object included in the image as shown in reference numeral <810>, and at least one feature area, such as an eye area 811 and a nose area 813, from the recognized face. ), and/or mouth region 815.

In one embodiment, the processor may receive voice data, such as “smells delicious” 817 from the user. The processor may include at least one graphic generated based on voice data such as “smells delicious” 817 as shown in reference numeral <820>, such as attributes (eg, font size, style, color, thickness, and/or The first graphic object 821, the second graphic object 823, and the third graphic object 825 to which different motions are applied may be displayed on the display.

In one embodiment, the processor detects a user input 827 for selecting a first graphic object 821 from among the displayed first graphic object 821, the second graphic object 823, and the third graphic object 825. can do. In response to detecting that the user input 827 for selecting the first graphic object 821 is detected, the processor may determine a position in the image to place the selected first graphic object 821 as shown in reference numeral <830>. have. For example, the processor may analyze the utterance content of the voice data, and determine a position to place the selected first graphic object 821 in the image. For example, the processor analyzes the voice data “smells delicious” 817, and “smells” is placed 831 in the feature region, eg, nose region 813, and “delicious” is the feature region, eg, mouth region 815. It can be decided to be placed 833.

The present invention is not limited thereto, and in another embodiment, the processor may receive an input for setting a position in an image in which a selected graphic object is to be placed.

In one embodiment, the processor may receive inputs 841 and 843 for setting the motion of the first graphic object 821, for example, “smells” and “delicious”, as shown at reference numeral <840>, , Based on the inputs 841 and 843, the motion of each of the first graphic object 821, for example, “smells” and “delicious” may be set. The processor images the first graphic object 821 on the basis of the motions 851 and 853 set in the first graphic object 821, for example, “smells” and “delicious”, as shown in reference numeral <850>. Can be synthesized in.

9A and 9B are diagrams for explaining a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments (900) )to be.

Referring to FIG. 9A, the processor (eg, the processor 270 of FIG. 2) includes the first face 913 and the second object of the first object 911 included in the image as shown in reference numeral <910>. The second face 917 of 915 may be recognized. The processor may include at least one feature area, for example, an eye area, a nose area, from the second face 917 of the first face 913 and the second face 917 recognized as illustrated by reference numeral <920>. And/or extracting the mouth region, and determining emotion information for the second object 915 based on the extracted feature region. For example, emotion information determined based on the extracted feature region is described assuming “anger”.

In one embodiment, the processor may receive first voice data, such as “what are you doing” 925. The processor may display at least one of the second face 917 of the second object 915 based on the received first voice data, for example, “what are you doing” 925 as shown in reference numeral <930>. A graphic object 931 may be generated and displayed. For example, the at least one graphic object may include text for the received voice data. The processor may determine emotion information on the first voice data based on at least one of the utterance content, sound tone, pitch, frequency, and sound pressure of the received first voice data. For example, the emotion information for the first voice data will be described assuming “tension” and “angry”. Determine the attribute of the text (eg, font size, style, color, thickness, or motion, or at least one of motion) based on the emotion information (eg, tension, anger) for the first voice data, and determine The text “what are you doing” with attribute applied (931) may be displayed.

In one embodiment, the processor is configured to generate second voice data such as “nothing… ”935. For example, the second voice data may be received after the first voice data is received and a designated time has elapsed. The processor may determine emotion information for the second voice data based on at least one of the utterance content, acoustic tone, pitch, frequency, and sound pressure of the received second voice data. For example, the emotion information for the second voice data is described assuming “worried”. As shown in reference numeral <940>, the processor determines the attribute of the text for the text for the second speech data based on the emotion information (for example, worry) about the second speech data, and the text to which the determined attribute is applied” nothing… ”941 can be displayed.

In one embodiment, the processor generates at least one graphic object (eg, text and/or stickers) among at least one feature area based on the utterance content of the voice data and the determined emotion information, as indicated by reference numeral <950>. ) To determine the region in the image to be synthesized. The processor may include at least one graphic object (eg, text “what are you doing” for the first speech data) 931, text “nothing…” for the second speech data (941), and/or The sticker 951 corresponding to the emotion information determined based on the feature region for the second object 915 may be synthesized.

In FIG. 9A according to an embodiment, it has been described that voice data, for example, first voice data and second voice data are received from a user at designated time intervals, but is not limited thereto. For example, the processor may simultaneously receive voice data from each of a plurality of users.

With reference to the embodiment of simultaneously receiving voice data from each of the plurality of users described above, in FIG. 9B to be described later, it will be described in detail.

Referring to FIG. 9B, the processor displays the first face 963 of the first object 961 and the second face 957 of the second object 965 included in the image as shown in reference numeral <960>. Can be recognized. The processor may include at least one feature region (eg, eye region, nose region, and/or mouth region) extracted from the recognized first face 963 and second face 967 as shown in reference numeral <970>. After determining the emotion information for the first object 961 and the second object 965 based on the basis, it is possible to receive a plurality of voice data from a plurality of users. For example, the processor receives the first voice data “what are you doing” 973 from a plurality of users, for example, the first user, and the second voice data “nothing…” from the second user. ”(977). The processor may determine emotion information for the first voice data and the second voice data based on at least one of the utterance content, sound tone, pitch, frequency, and sound pressure of the received first voice data and the second voice data.

In one embodiment, the processor includes first voice data such as “what are you doing” 973 and second voice data such as “nothing… Based on ”977, at least one graphic object may be generated and displayed. For example, the at least one graphic object is text for the first voice data and the second voice data to which the attribute of the text determined based on emotion information (eg, at least one of font size, style, color, thickness, or motion) is applied. It may include.

In one embodiment, the processor may compare emotion information for the first object 961 and the second object 965 with emotion information for the first voice data and the second voice data. For example, the operation of comparing emotion information may be an operation of analyzing whether similarity of emotion information and/or emotion information of the same category (eg, positive emotion information, negative emotion information). Based on the comparison result, the processor may determine an area in the image in which at least one graphic object generated based on the first audio data and the second audio data is to be placed. For example, emotion information for the first object 961 is “sadness”, emotion information for the second object 965 is “angry”, and emotion information for the first voice data is “angry” and the second voice data. Assuming that the emotion information for the user is “worried”, the processor determines the text for the second voice data having the emotion information “worried” as an area in which the feature area of the first object 961 having the emotion information “sadness” will be arranged. Then, the text for the first voice data having the emotion information “angry” can be determined as an area in which the feature region of the second object 965 having the emotion information “angry” is arranged.

In one embodiment, the processor may include at least one graphic object (eg, text “what are you doing” for the first voice data) (981) and a second voice generated in the determined area as shown at reference numeral <980>. The text “nothing…” 983 for data and/or the sticker 985 corresponding to the emotion information determined based on the feature area for the second object 965 may be synthesized and displayed.

10 is a diagram 1000 illustrating a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.

Referring to FIG. 10, the processor may remove a plurality of objects included in an image, for example, the first face 1013 and the second object 1015 of the first object 1011 as shown in reference numeral <1010>. 2 Face 1017 can be recognized.

In one embodiment, the processor may receive voice data, eg, “welcome back” 1019. The processor may generate at least one graphic object based on the received voice data, for example, “welcome back” 1019 as shown in reference numeral <1020> and display 1021 on the display. For example, the at least one graphic object may include text for the received voice data. The text for the voice data may have attributes determined based on at least one of speech content, acoustic tone, pitch, frequency, and sound pressure of the received voice data. For example, emotion information determined based on voice data will be described assuming “happy”.

In one embodiment, the processor may separate text for speech data based on the number of recognized objects, as indicated by reference numeral <1030>. For example, text may be separated into letters constituting a word. For example, the graphic object 1031 to be placed on the first face 1013 of the first object 1011 is “welcome”, and the graphic object 1033 to be placed on the second face 1017 of the second object 1015 Can be determined as “back”.

In one embodiment, the processor provides at least one graphic object, eg, stickers 1041, 103, 1045, corresponding to emotion information, for example, “happiness” determined based on voice data as shown at reference numeral <1040>. It can be synthesized on the first face 1013 of one object 1011 and the second face 1017 of the second object 1015.

In one embodiment, the processor displays each of the graphic objects 1031 “welcome” and the graphic objects 1033 “back” together with the stickers 1041, 1043, 1045 as shown in reference numeral <1050>, for example, The first face 1013 of the first object 1011 and the second face 1017 of the second object 1015 may be synthesized.

11 is a diagram 1100 illustrating a signal flow between electronic devices that provide a corresponding graphic object corresponding to emotion information, according to various embodiments.

In the embodiment according to FIG. 11, the performance of the external electronic device 1120 may be higher than that of the electronic device 1110, and the high performance external electronic device 1120 analyzes information related to an image and analyzes the results Can be provided to the electronic device 1110.

Referring to FIG. 11, the electronic device 1110 may display an image in operation 1130. In one embodiment, the image may include at least one image photographed from at least one camera (eg, the camera 220 of FIG. 2 ), and an image stored in a memory (eg, the memory 230 of FIG. 2 ). .

In one embodiment, the electronic device 1110 may determine information related to an image to be analyzed in operation 1135. For example, the information related to the image to be analyzed is a second image different from the image displayed in operation 1130, audio information included in the image, text information included in the image, biometric information of the user of the electronic device 1110, or context information It may include at least one of.

For example, the second image is an image including an image displayed in operation 1130 (eg, one of a frame image of the image, an image including at least a part of a face, body, and object in the first image), a barcode, and a two-dimensional barcode ( Examples: QR code, data matrix code), markers, or text images. The audio information may include audio track information of audio, sound signals, and/or images received by the electronic device 1110. Biometric information is biometric information (eg, HRM, HRV, SPO2, blood pressure, blood sugar, EEG, skin resistance, electromyogram, electrocardiogram, respiratory rate per minute, or body temperature) received through a biometric sensor of the electronic device 1110, electronic device Biometric information (eg, blood flow change image) obtained by analyzing an image through a camera of 1110, and/or body pose or body motion information by a motion sensor may be included. The text information may include text input by a pen input, a virtual keyboard, or a keypad, a symbol, a symbol, a caption track of an image, associated caption file information, or subtitle information included or associated with music content. The context information includes current time, current location, and schedule related information (eg, birthday, Christmas) for the electronic device 1110, profile information (eg, preference, food, personal information) for the user of the electronic device 1110 (Eg, gender, age, visit information). In one embodiment, emotion analysis target information may be included in an image or may be separately received (eg, voice data).

In one embodiment, the electronic device 1110 may transmit information related to an image to be analyzed to the external electronic device 1120 in operation 1140.

In one embodiment, the external electronic device 1120 analyzes information related to an image received from the electronic device 1110 in operation 1145 and determines emotion information based on the information. For example, the external electronic device 1120 is based on at least one of a second image different from the displayed image, audio information included in the image, text information included in the image, biometric information of the user of the electronic device 1110, or context information. Emotion information.

In one embodiment, the external electronic device 1120 may determine at least one graphic object based on the determined emotion information in operation 1150. In another embodiment, the at least one graphic object may be determined based on the preferences of the user of the electronic device 1110 or another user.

In one embodiment, the external electronic device 1120 may transmit the determined emotion information and at least one graphic object to the electronic device 1110 in operation 1155. In operation 1160, the electronic device 1110 may determine a position to synthesize at least one graphic object in the image. In operation 1165, the electronic device 1110 may determine a graphic object to be synthesized in an image among the received at least one graphic object, and in operation 1170, the determined graphic object may be synthesized at the determined location.

12 is a diagram 1200 illustrating signal flow between electronic devices that provide a corresponding graphic object corresponding to emotion information according to various embodiments.

In the embodiment according to FIG. 12, the first external electronic device 1210 receives analysis target information and provides the received analysis target information to the electronic device 1220 (eg, the electronic device 1110 of FIG. 11 ). Can be. For example, the first external electronic device 1210 may include a smart watch, head mounted display (HMD), augmented reality glass (AR glass), drone, robot, and Internet of Things (IoT) device. In one embodiment, the first external electronic device 1210 may recognize and process image or audio information, and may also receive and transmit the image to the electronic device 1220.

In one embodiment, the performance of the second external electronic device 1230 (eg, the first external electronic device 1120 of FIG. 11) is higher than that of the electronic device 1220 and the first external electronic device 1210. The second external electronic device 1230 having high performance may analyze information related to an image and provide the analyzed result to the electronic device 1220. For example, information related to an image may include an image, audio, sound-to-text (STT), barcode, marker, and emotion information (or emotion, mood). In one embodiment, the second external electronic device 1230 may manage preference information based on a user profile received from electronic devices and provide a graphic object based on the preference information. The second external electronic device 1230 may store and manage graphic objects and transmit the graphic object to the electronic device 1220. When a request is received from the electronic device 1220, the determined candidate graphic object and image may be received to perform synthesis and editing operations. For example, the second external electronic device 1230 includes a server, a smart home hub device, a cloud computing device, a mobile edge comp. (MEC), an access point device, an Internet message server (IMS), and multimedia streaming It may include either a server or a video and video communication host device.

Referring to FIG. 12, in operation 1241, the first external electronic device 1210 may obtain context information about the first external electronic device 1210. For example, the context information may include image information, audio information, text data, and/or biometric information obtained from the first external electronic device 1210.

In one embodiment, the electronic device 1220 may display an image in operation 1243.

In one embodiment, the first external electronic device 1210 may transmit the obtained context information to the electronic device 1220 in operation 1245.

In an embodiment, the electronic device 1220 may analyze at least one of at least one feature region extracted from the image, data related to the image, and context information obtained from the first external electronic device 1210 in operation 1247. have. In operation 1249, the electronic device 1220 may transmit the analysis result to the second external electronic device 1230.

In one embodiment, the second external electronic device 1230 determines the emotion information based on the received analysis result in operation 1251.

The electronic device 1220 according to an embodiment analyzes at least one of at least one feature region extracted from an image, data related to the image, and context information obtained from the first external electronic device 1210 and then analyzes the result. It has been described as transmitting to the second external electronic device 1230, but is not limited thereto. The electronic device 1220 determines at least one of at least one feature region extracted from the image, data related to the image, and context information obtained from the first external electronic device 1210 as information related to the image to be analyzed and removes it. 2 to the external electronic device 1230. The second external electronic device 1230 analyzes information related to an image received from the electronic device 1220 and determines emotion information based on the information.

In one embodiment, the second external electronic device 1230 may transmit the determined emotion information to the electronic device 1220 in operation 1253.

In one embodiment, in operation 1255, the electronic device 1220 may determine a location to display the graphic object in the image. In operation 1257, the electronic device 1220 may request the second external electronic device 1230 a graphic object to be synthesized in the image.

In one embodiment, the second external electronic device 1230 determines at least one graphic object based on the determined emotion information in operation 1259, and in step 1261, determines the determined at least one graphic object to the electronic device 1220. Can transmit.

In one embodiment, in operation 1263, the electronic device 1220 may determine a graphic object to be synthesized in an image among at least one graphic object received from the second external electronic device 1230. In operation 1265, the electronic device 1220 may request the second external electronic device 1230 to synthesize the determined graphic object in the image.

In one embodiment, the second external electronic device 1230 may synthesize the graphic object determined at the determined location in the image in operation 1267, and transmit the synthesized graphic object image to the electronic device 1220 in operation 1269. .

In one embodiment, the electronic device 1220 may receive and display a composite image of a graphic object from the second external electronic device 1230 in operation 1271.

According to the above-described embodiments of FIGS. 3 to 12, the processor may include at least one feature region extracted for at least one object recognized in the image, data related to the image, context information, or information related to the analyzed voice data. Emotion information may be determined based on at least one, and at least one graphic object may be generated based on the determined emotion information. The processor may synthesize at least one object generated based on emotion information into an image and store it in a memory so that it can communicate with other users. As the text applied with attributes corresponding to at least one feature area, image-related data and context information as well as emotion information related to received voice data is synthesized, the user's emotion can be more richly transmitted to other users.

The electronic device according to various embodiments disclosed in the present document may be various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

It should be understood that various embodiments of the document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and include various modifications, equivalents, or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and Each of the phrases such as “at least one of A, B, or C” may include any of the items listed together in the corresponding phrase of the phrases, or any possible combination thereof. Terms such as “first”, “second”, or “first” or “second” can be used to simply distinguish a component from other components, and to separate components from other aspects (eg, importance or Order). Any (eg first) component is referred to as a “coupled” or “connected” to another (eg second) component, with or without the term “functionally” or “communically” If mentioned, it means that any of the above components can be connected directly to the other components (eg by wire), wirelessly, or through a third component.

As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic blocks, components, or circuits. The module may be an integrally configured component or a minimum unit of the component or a part thereof performing one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, processor 120) of a device (eg, electronic device 101) may call and execute at least one of one or more commands stored from a storage medium. This enables the device to be operated to perform at least one function according to the at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The storage medium readable by the device may be provided in the form of a non-transitory storage medium. Here,'non-transitory' only means that the storage medium is a tangible device, and does not include a signal (eg, electromagnetic waves), and this term is used when data is stored semi-permanently in a storage medium. It does not distinguish between temporary storage cases.

According to an embodiment, a method according to various embodiments disclosed in this document may be provided as being included in a computer program product. Computer program products can be traded between sellers and buyers as products. The computer program product may be distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play StoreTM) or two user devices ( For example, it can be distributed directly between smartphones) and online (eg, downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product may be stored at least temporarily on a storage medium readable by a device such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, modules or programs) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, or omitted Or, one or more other actions can be added.

210: wireless communication circuit 220: camera
230: memory 240: sensor unit
250: audio processing circuit 260: touch screen display
270: processor

Claims (20)

In the electronic device,
display;
MIC;
A processor operatively connected to the display and the microphone; And
It includes a memory operatively connected to the processor,
When the memory is executed, the processor,
Displaying an image on the display,
Recognize at least one object from the displayed image,
Extracting at least one feature region for the recognized at least one object,
When at least one voice data is received through the microphone, first emotion information is determined based on the received at least one voice data,
Generating at least one graphic object related to the at least one voice data based on the first emotion information,
Determine an area to synthesize the generated at least one graphic object among the at least one feature area based on the utterance content of the at least one voice data, and
An electronic device that stores instructions for synthesizing the generated at least one graphic object at a position corresponding to the determined area. According to claim 1,
The instructions, the processor,
Determine second emotion information for the at least one object based on at least one of the extracted at least one feature region, the image-related data, or context information, and
An electronic device configured to generate at least one graphic object associated with the second emotion information. According to claim 2,
The instructions, the processor,
Determine third emotion information based on the first emotion information and the second emotion information, and
An electronic device configured to generate at least one graphic object related to the third emotion information based on the determined third emotion information. According to claim 2,
The instructions, the processor,
Displaying at least one graphic object associated with second emotion information on the generated at least one object on the display, and
An electronic device configured to synthesize the selected graphic object on the image based on a user input for selecting one of the at least one graphic object associated with the displayed second emotion information. According to claim 2,
The instructions, the processor,
If a priority is set for the at least one feature region, the data related to the image, or the context information, based on the information having a higher priority among the at least one feature region, the data related to the image, or the context information An electronic device configured to determine second emotion information for the at least one object. According to claim 1,
The instructions, the processor,
When the at least one voice data is received, to analyze at least one of the speech content, the user's voice tone, pitch, frequency, or sound pressure included in each of the voice data, for each of the received at least one voice data Electronic devices. According to claim 1,
At least one graphic object related to the at least one voice data generated based on the first emotion information includes text to which the attribute corresponding to the determined first emotion information is applied,
The attribute includes at least one of a font size, style, color, thickness, or motion corresponding to the first emotion information. The method of claim 7,
The instructions, the processor,
When the text to which the attribute is applied is plural, the text to which the plural attribute is applied is displayed on the display, and
When a user input for selecting one of the texts to which the displayed plurality of attributes is applied is detected, the electronic device is configured to synthesize the selected text at a position corresponding to the determined area. According to claim 1,
The instructions, the processor,
An electronic device that stores the synthesized image of the at least one graphic object in the memory. According to claim 2,
The data related to the image includes at least one of a background of the image, a type of an object disposed in the foreground of the image, a color of the image, audio included in the image, or text,
The situation information includes at least one of a current time, a current location, or an event schedule. In the electronic device,
display;
A processor operatively connected to the display; And
It includes a memory operatively connected to the processor,
When the memory is executed, the processor,
Displaying an image on the display,
Recognize at least one object from the displayed image,
Emotion information is determined based on at least one of data related to the image, context information, or information related to a user of the electronic device,
When an area in the image to synthesize at least one graphic object corresponding to the determined emotion information is determined, the display displays one or more graphic objects corresponding to the determined emotion information to be synthesized in the area based on characteristics of the determined area. Marked on, and
When the user input for selecting at least one graphic object among the displayed one or more graphic objects is detected, the electronic device stores instructions for synthesizing the selected at least one graphic object at a position corresponding to the determined area. Device. The method of claim 11,
The instructions, the processor,
An electronic device that allows the provided one or more graphic objects to be provided differently based on characteristics of an area in the image. The method of claim 11,
The instructions, the processor,
An electronic device for determining an area in the image to synthesize the at least one graphic object based on a user input for selecting an area to synthesize the at least one graphic object in the image. In the method of providing a graphic object corresponding to the emotion information of the electronic device,
Displaying an image on a display;
Recognizing at least one object from the displayed image;
Extracting at least one feature region for the recognized at least one object;
If at least one voice data is received through the microphone, determining first emotion information based on the received at least one voice data;
Generating at least one graphic object related to the at least one voice data based on the first emotion information;
Determining an area to synthesize the generated at least one graphic object among the at least one feature area based on the speech content of the at least one voice data; And
And synthesizing the generated at least one graphic object at a position corresponding to the determined area. The method of claim 14,
After extracting the at least one feature region, determining second emotion information for the at least one object based on at least one of the extracted at least one feature region, the image-related data, or context information; And
And generating at least one graphic object associated with the second emotion information. The method of claim 15,
Determining third emotion information based on the first emotion information and the second emotion information; And
And generating at least one graphic object related to the third emotion information based on the determined third emotion information. The method of claim 14,
When the at least one voice data is received, an operation of analyzing at least one of the speech content, the user's voice tone, pitch, frequency, or sound pressure included in the voice data for each of the received at least one voice data How to include more. The method of claim 14,
At least one graphic object related to the at least one voice data generated based on the first emotion information includes text to which the attribute corresponding to the determined first emotion information is applied,
The attribute includes at least one of font size, style, color, thickness, or motion corresponding to the first emotion information. In the method of providing a graphic object corresponding to the emotion information of the electronic device,
Displaying an image on a display;
Recognizing at least one object from the displayed image;
Determining emotion information based on at least one of data related to the image, context information, or information related to a user of the electronic device;
When an area in the image to synthesize at least one graphic object corresponding to the determined emotion information is determined, the display displays one or more graphic objects corresponding to the determined emotion information to be synthesized in the area based on characteristics of the determined area. Operation to display on; And
And when a user input for selecting at least one graphic object among the displayed one or more graphic objects is detected, synthesizing the selected at least one graphic object at a position corresponding to the determined area. The method of claim 19,
The operation of displaying the one or more graphic objects on the display,
And providing the one or more provided graphic objects differently based on the characteristics of the region in the image.

Images