KR102667547B1 - Electronic device and method for providing graphic object corresponding to emotion information thereof - Google Patents

Abstract

According to various embodiments of the present invention, an electronic device includes a display, a microphone, a processor operatively connected to the display and the microphone, and a memory operatively connected to the processor, wherein the memory is configured to operate when executed. , the processor displays an image on the display, recognizes at least one object from the displayed image, extracts at least one feature area for the at least one recognized object, and transmits at least one feature through the microphone. When voice data is received, determine first emotion information based on the at least one received voice data, and generate at least one graphic object related to the at least one voice data based on the first emotion information, Based on the speech content of the at least one voice data, determine an area in which to synthesize the generated at least one graphic object among the at least one feature area, and place the generated at least one graphic object at a position corresponding to the determined area. Instructions for compositing graphic objects can be stored.
Various other embodiments other than those disclosed in the present invention are possible.

Description

Electronic device and method of providing graphic objects corresponding to emotional information using the same {ELECTRONIC DEVICE AND METHOD FOR PROVIDING GRAPHIC OBJECT CORRESPONDING TO EMOTION INFORMATION THEREOF}

Various embodiments of the present invention relate to a method of providing graphic objects corresponding to emotional information.

Recently, online communities such as social network services (SNS) and messenger services have become active, allowing users to share their daily lives with others online. For example, a user writes down his or her daily life or thoughts and shares them with the other person, and along with the text, emoticons and emojis predefined based on the facial feature area extracted from the user image (e.g., hairstyle, glasses, hat) , stickers) can be applied to express one's feelings to the other person directly or indirectly.

Electronic devices provide predefined emoticons or emojis to users, but there are limitations in expressing the user's emotions because the types of predefined emoticons or emojis are limited. Additionally, in an augmented reality (AR) environment, users cannot photograph their own faces and thus cannot convey their emotions to the other person.

An electronic device according to various embodiments of the present invention may determine emotional information of an object based on a feature area of the object included in an image and provide a graphic object corresponding to the determined emotional information.

Electronic devices according to various embodiments of the present invention may determine emotional information based on input voice data, map it to a recognized object in an image, and provide a graphic object related to voice data corresponding to the emotional information.

Electronic devices according to various embodiments of the present invention can apply the user's emotions to other objects (eg, animals or objects) obtained from a camera in an AR environment where the user's own face cannot be photographed.

According to various embodiments of the present invention, an electronic device includes a display, a microphone, a processor operatively connected to the display and the microphone, and a memory operatively connected to the processor, wherein the memory is configured to operate when executed. , the processor displays an image on the display, recognizes at least one object from the displayed image, extracts at least one feature area for the at least one recognized object, and transmits at least one feature through the microphone. When voice data is received, determine first emotion information based on the at least one received voice data, and generate at least one graphic object related to the at least one voice data based on the first emotion information, Based on the speech content of the at least one voice data, determine an area in which to synthesize the generated at least one graphic object among the at least one feature area, and place the generated at least one graphic object at a position corresponding to the determined area. Instructions for compositing graphic objects can be stored.

According to various embodiments of the present invention, an electronic device includes a display, a processor operatively connected to the display, and a memory operatively connected to the processor, wherein the memory, when executed, allows the processor to: Displaying an image on the display, recognizing at least one object from the displayed image, and determining emotional information based on at least one of data related to the image, situational information, or information related to a user of the electronic device, When a region within the image in which to composite at least one graphic object corresponding to the determined emotional information is determined, displaying one or more graphic objects corresponding to the determined emotional information to be composited in the region based on characteristics of the determined region. and instructions for compositing the selected at least one graphic object at a position corresponding to the determined area when a user input for selecting at least one graphic object among the displayed one or more graphic objects is detected. ) can be saved.

According to various embodiments of the present invention, a method of providing a graphic object corresponding to emotional information of an electronic device includes: displaying an image on a display, recognizing at least one object from the displayed image, and recognizing the recognized object. Extracting at least one feature area for at least one object, when at least one voice data is received through the microphone, determining first emotion information based on the at least one received voice data, An operation of generating at least one graphic object related to the at least one voice data based on first emotion information, the generated at least one of the at least one feature area based on the speech content of the at least one voice data It may include an operation of determining an area in which to synthesize a graphic object, and an operation of combining the at least one generated graphic object at a position corresponding to the determined area.

According to various embodiments of the present invention, a method of providing a graphic object corresponding to emotional information of an electronic device includes an operation of displaying an image on a display, an operation of recognizing at least one object from the displayed image, the image and An operation of determining emotional information based on at least one of related data, situational information, or information related to a user of the electronic device, when an area within the image in which to synthesize at least one graphic object corresponding to the determined emotional information is determined. , displaying on the display one or more graphic objects corresponding to the determined emotional information to be synthesized in the region based on the characteristics of the determined region, and selecting at least one graphic object among the displayed one or more graphic objects. When a user input is detected, the method may include synthesizing the selected at least one graphic object at a position corresponding to the determined area.

An electronic device according to various embodiments of the present invention may determine emotional information about an image based on data related to the image or input audio data, and synthesize a graphic object corresponding to the determined emotional information into the image. Users can communicate with the other person using images synthesized with graphic objects corresponding to emotional information, allowing the user to convey the user's emotions to the other person more richly.

The electronic device according to various embodiments of the present invention reinterprets the user's emotions in an AR environment where the user's own face cannot be photographed according to the human characteristic area of an object (or animal) obtained from a camera. It can provide fun elements to users.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
FIG. 2 is a block diagram illustrating an electronic device according to various embodiments.
FIG. 3 is a flowchart illustrating a method of providing a graphic object corresponding to emotional information, according to various embodiments.
FIG. 4 is a flowchart illustrating a method of providing a graphic object corresponding to emotional information, according to various embodiments.
FIG. 5 is a diagram illustrating a method of determining emotional information by extracting a feature area for at least one object recognized from an image, according to various embodiments.
FIG. 6 is a diagram illustrating a method of generating at least one graphic object based on determined emotional information, according to various embodiments.
FIG. 7 is a diagram illustrating a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.
FIG. 8 is a diagram illustrating a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.
FIGS. 9A and 9B are diagrams for explaining a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.
FIG. 10 is a diagram illustrating a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.
FIG. 11 is a diagram illustrating signal flow between electronic devices that provide graphic objects corresponding to emotional information, according to various embodiments.
FIG. 12 is a diagram illustrating signal flow between electronic devices that provide corresponding graphic objects corresponding to emotional information, according to various embodiments.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments.

Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176), interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) may include. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted, or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes a main processor 121 (e.g., a central processing unit or an application processor), and an auxiliary processor 123 (e.g., a graphics processing unit, an image signal processor) that can operate independently or together with the main processor 121. , sensor hub processor, or communication processor). Additionally or alternatively, the auxiliary processor 123 may be set to use less power than the main processor 121 or to specialize in a designated function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), along with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display device 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions associated therewith. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or an application 146.

The input device 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, stylus pen).

The sound output device 155 may output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display device 160 may include touch circuitry configured to detect a touch, or a sensor circuit configured to measure the intensity of force generated by the touch (e.g., a pressure sensor). there is.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, WiFi direct, or IrDA (infrared data association)) or a second network 199 (e.g., a cellular network, the Internet, Alternatively, it may communicate with an external electronic device through a computer network (e.g., a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 to communicate within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed and authenticated.

The antenna module 197 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module 197 may include one antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, connected to the plurality of antennas by the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, RFIC) in addition to the radiator may be additionally formed as part of the antenna module 197.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be the same or different type of device from the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 must perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technologies may be used.

FIG. 2 is a block diagram 200 showing an electronic device according to various embodiments.

Referring to FIG. 2, the electronic device 201 (e.g., the electronic device 101 of FIG. 1) includes a wireless communication circuit 210 (e.g., the communication module 190 of FIG. 1) and a camera 220 (e.g., Camera module 180 in FIG. 1), memory 230 (e.g., memory 130 in FIG. 1), sensor unit 240 (e.g., sensor module 176 in FIG. 1), audio processing circuit 250 (e.g., audio module 170 in FIG. 1), a touch screen display 260 (e.g., display device 160 in FIG. 1), and a processor 270 (e.g., processor 120 in FIG. 1). can do.

According to one embodiment, the wireless communication circuit 210 (e.g., communication module 190 of FIG. 1) connects an electronic device 201 (e.g., electronic device 101 of FIG. 1) and an external electronic device (e.g., Communication between the electronic device 102 and the electronic device 104 of FIG. 1) or a server (eg, the server 108 of FIG. 1) may be connected.

According to one embodiment, the camera 220 (e.g., the camera module 180 in FIG. 1) transmits the collected image as a preview image to the display 261 so that the user can view the image projected through the camera 220. You can check it. In response to an input requesting capture, the camera 220 may generate image data by capturing images collected at the time the input requesting capture occurs. In one embodiment, one or more cameras 220 may be provided.

According to one embodiment, the memory 230 (e.g., the memory 130 in FIG. 1) includes a program for generating and combining a graphic object corresponding to emotional information into an image and a program according to embodiments of the present invention, the above An image composed of graphic objects can be saved. The memory 230 may store a standard value for determining emotional information based on information related to the feature area, data in the image, situation information, and voice data extracted from the object. The memory 230 may store at least one graphic object corresponding to emotional information. The memory 230 may store attribute values corresponding to emotional information to be applied to text for voice data. The memory 230 may store an image in which at least one graphic corresponding to emotional information is synthesized.

According to one embodiment, the sensor unit 240 (e.g., the sensor module 176 in FIG. 1) detects the external environmental state (e.g., user state) of the electronic device 201 and generates electricity corresponding to the detected state. It can generate signals or data values. For example, the sensor unit 240 may include a gesture sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

According to one embodiment, the audio processing circuit 250 (e.g., the audio module 170 of FIG. 1) may include a speaker and a microphone, although not shown. The audio processing circuit 250 may further include earphones and a microphone. In one embodiment, the microphone may receive sound (e.g., voice data), convert it into an electrical signal, and provide it to the processor 270.

According to one embodiment, the touch screen display 260 (eg, the display device 160 of FIG. 1) may be configured as an integrated unit including the display 261 and the touch panel 263.

According to one embodiment, the touch screen display 260 may display an image under the control of the processor 270. The touch screen display 260 may display at least one graphic object corresponding to emotional information generated under the control of the processor 270. The touch screen display 260 may synthesize at least one object created in a specific area in an image under the control of the processor 270 and display an image in which the at least one object is synthesized.

According to one embodiment, the processor 270 (e.g., processor 120 in FIG. 1) controls the overall operation of the electronic device 201 and signal flow between internal components of the electronic device 201, and processes data. and control the power supply from the battery to the components.

According to one embodiment, the processor 270 may display an image on the display 261. For example, the displayed image may include at least one image acquired from at least one camera 220, or an image stored in the memory 230. The processor 270 may recognize at least one object from the displayed image. there is. For example, at least one object may include at least one of a person, an animal, or an object. The processor 270 may extract at least one feature area (eg, a facial feature area including an eye area, a nose area, and/or a mouth area) for at least one recognized object.

In one embodiment, the processor 270 may determine emotional information based on at least one of at least one extracted feature area, data related to an image, or situation information. For example, data related to an image includes the background of the image, the object placed in the foreground, the type of object (e.g. gun, knife, sharp object), the color of the image (e.g. saturation, brightness), and the contents of the image. Embedded audio and/or embedded text may be included in the video, and contextual information may include current time, current location, and/or event schedule.

In one embodiment, when at least one voice data is received from at least one user through a microphone (not shown), the processor 270 stores the speech content for the at least one received voice data and each of the at least one voice data. At least one of the included user's voice tone, pitch, frequency, or sound pressure can be analyzed. The processor 270 may determine emotional information for each at least one piece of voice data based on the analyzed information. The processor 270 may generate at least one graphic object to be synthesized into an image based on emotional information for each of the determined at least one voice data. For example, at least one graphic object may include a sticker corresponding to the determined emotional information, an emoji, and/or text to which an attribute related to at least one voice data is applied.

In one embodiment, the processor 270 determines an area in which to synthesize at least one graphic object generated among at least one feature area based on the speech content of at least one voice data, and determines an area in which to synthesize at least one graphic object generated at a position in the determined image. You can composite a single graphic object. The processor 270 may store an image composed of at least one graphic object in the memory 230 .

FIG. 3 is a flowchart 300 illustrating a method of providing a graphic object corresponding to emotional information, according to various embodiments.

Referring to FIG. 3, the processor (e.g., processor 270 of FIG. 2) may display an image on a display (e.g., display 261 of FIG. 2) in operation 301. For example, the displayed image may include at least one image acquired from at least one camera (e.g., camera 220 in FIG. 2) or an image stored in memory (e.g., memory 230 in FIG. 2). .

In one embodiment, the processor may recognize at least one object from the displayed image in operation 303. For example, at least one object may include at least one of a person, an animal, or an object. In one embodiment, at least one object may be recognized automatically or may be recognized through a user input that selects at least one object in an image.

In one embodiment, the processor may extract at least one feature area for at least one recognized object in operation 305. For example, the at least one feature area may include, for example, a facial feature area, an eye area, a nose area, and/or a mouth area.

In one embodiment, at least one feature area may be extracted automatically or may be extracted through a user input that selects at least one feature area. A control point (eg, marker) may be created on at least one extracted feature area, such as the outline of the left and right eye areas, the outline of the nose area, and/or the outline of the mouth area. In another embodiment, the processor may detect a user input that adjusts a control point created on the outline of at least one extracted feature area, and may adjust the at least one extracted feature area based on the detected user input. At least one feature area for at least one object can be accurately extracted through user input for adjusting the control point.

In one embodiment, the processor may further perform an operation of tracking a change in at least one feature area for at least one object based on the difference image between image frames. For example, a change in at least one feature area, such as eye movement or mouth movement, may be used to determine the first emotional information of operation 307, which will be described later.

In one embodiment, in operation 307, the processor may determine the first emotion information based on at least one of the extracted feature area, data related to the image, or situation information. For example, the first emotion information may include at least one of happiness, anger, worry, sadness, boredom, disgust, or daily life.

In one embodiment, the processor may determine the first emotion information based on at least one extracted feature area, for example, the distance between the left and right eyes and/or the shape of the mouth. Without being limited thereto, the processor may determine the first emotional information based on the movement of at least one feature area, such as eye movement, lip movement, and/or facial movement.

In one embodiment, data related to the image includes the background of the image, the object placed in the foreground, the type of object (e.g., gun, knife, sharp object), and the color of the image (e.g., saturation, brightness). , audio included in the video, and/or text included in the video. The processor may determine first emotion information based on data related to the image.

In one embodiment, contextual information may include current time, current location, and/or event schedule. The processor may determine the first emotional information based on the situation information. For example, the first emotional information may be determined by analyzing events according to the current time. If the time-dependent event is a birthday or Christmas, the first emotional information may be determined to be positive emotional information (e.g., happiness), and if the time-dependent event is Halloween, the first emotional information may be determined to be negative emotional information (e.g., happiness). You can decide by fear. Alternatively, based on the current location, for example, if it is an amusement park, the first emotional information may be determined as positive emotional information (e.g., happiness), and if the current location is a haunted house, the first emotional information may be determined as negative emotional information (e.g., happiness). You can decide by fear.

In another embodiment, priority may be set to at least one feature area, data related to an image, or context information. The processor may determine the first emotion information based on high priority information among at least one feature area, image-related data, or situation information.

In one embodiment, the processor may receive at least one voice data through a microphone (e.g., audio processing circuit 250 of FIG. 2) in operation 309. For example, at least one voice data may be received from at least one user. In operation 311, the processor may analyze at least one received voice data and determine second emotion information for each of the at least one voice data based on the analysis. The second emotion information may include at least one of happiness, anger, worry, sadness, boredom, disgust, or daily life. For example, when at least one voice data is received, the processor may select at least one of the speech content for each of the at least one received voice data, the user's voice tone, pitch, frequency, or sound pressure included in each of the at least one voice data. It can be analyzed.

In one embodiment, in operation 313, the processor may generate at least one graphic object to be composited into an image based on at least one of first emotion information and second emotion information. For example, the at least one graphic object has a sticker corresponding to at least one of the first emotion information or the second emotion information, an emoji, and/or an attribute related to at least one voice data generated based on the second emotion information applied. Can contain text. For example, the text related to at least one voice data generated based on the second emotion information may include text to which at least one of properties corresponding to the second emotion information, such as font size, style, color, thickness, or motion, is applied. You can.

In another embodiment, when priorities are set for the first emotion information and the second emotion information, the processor may synthesize at least one emotion information into the image based on the emotion information with the higher priority among the first emotion information and the second emotion information. You can create graphic objects.

In one embodiment, when text to which an attribute corresponding to the second emotional information is applied is generated, the processor may display the text to which the attribute is applied on a display. In one embodiment, when there are a plurality of texts to which an attribute is applied, the processor may detect a user input for selecting one text among the plurality of texts. The processor can synthesize text selected by user input into an image.

In one embodiment, text to which attributes are applied may be customized by the user. At least one accurate graphic object corresponding to emotional information can be created through customization of text to which attributes are applied.

In one embodiment, in operation 315, the processor may determine an area in which to synthesize at least one generated graphic object among at least one feature area based on the speech content of at least one voice data. For example, if the utterance content of at least one voice data is mouth-related (e.g., “delicious”), the processor determines, for example, the mouth area among the at least one feature area as the area in which to synthesize at least one generated graphic object. You can. For another example, if the utterance content of at least one piece of speech data is content related to the mouth (e.g., “delicious”) and content related to the nose (e.g., “smells good”), the processor selects one of the at least one feature area, e.g. , the mouth area and nose area may be determined as areas to composite at least one generated graphic object.

However, the method is not limited to this, and an area to composite at least one graphic object may be determined based on a user input for selecting a position to place at least one graphic object in the image.

In one embodiment, in operation 317, the processor may synthesize at least one graphic object created at a location within the determined image. In operation 319, the processor may store an image in which at least one graphic object is synthesized in memory.

In FIG. 3 according to one embodiment, it has been described that operation 315 is performed after creating at least one graphic object in operation 313, but the operation is not limited thereto. After generating at least one graphic object in operation 313, the processor may further receive voice data from the user.

In one embodiment, it has been described as generating at least one graphic object to be composited into an image based on at least one of the first emotion information or the second emotion information in operation 313, but the present invention is not limited to this. For example, the processor may generate at least one graphic object based on the first emotion information for which the first emotion information of operation 307 is determined and provide a list thereof. After the second emotion information of the operation 311 is determined, the processor may generate at least one graphic object based on the second emotion information and provide a list thereof. The processor may determine third emotion information based on the first emotion information and the second emotion information. The processor may generate at least one graphic object related to the third emotion information based on the third emotion information and provide a list thereof. The user can select a graphic object to be synthesized in the video from the list corresponding to each emotional information. By fusing the first emotion information and the second emotion information, the third emotion information is determined and a graphic object for it is provided, thereby obtaining an optimal graphic object for the image and at least one object recognized within the image.

FIG. 4 is a flowchart 400 illustrating a method of providing a graphic object corresponding to emotional information, according to various embodiments.

Referring to FIG. 4, a processor (eg, processor 270 of FIG. 2) may display an image in operation 401. In operation 403, the processor may determine emotional information based on at least one of data related to the image, situation information, or information related to the user. For example, information related to an image may include audio information included in the image and/or movement information of at least one object included in the image. Contextual information may include current time, current location, and/or event schedule. Information related to the user may include at least one biometric information, at least one gesture information, and/or at least one voice data.

In one embodiment, in operation 405, the processor may determine an area within the image to composite at least one graphic object. For example, an area in which to place at least one graphic object may be determined based on a user input for selecting an area in which to composite at least one graphic object from an image. In operation 407, the processor may provide one or more graphic objects corresponding to emotional information to be synthesized based on the characteristics of the determined area. For example, one or more graphic objects may be provided differently based on the determined area within the image. If the determined area in the image is the eye area, eye images corresponding to emotional information can be provided. If the area within the determined image is an area where the animation character object is displayed, corresponding character images with emotional information reflected in the animation character object can be provided. If the area within the determined image is an area that displays the background image, the color, brightness, and/or transparency of the background image excluding the foreground may be determined based on the emotion information and provided.

In one embodiment, in operation 409, the processor may detect an input for selecting at least one graphic object to be composited into an image from among one or more provided graphic objects. In operation 411, the processor may synthesize at least one graphic object selected at a position within the determined image, and in operation 413, store the image in which the at least one graphic object is synthesized in the memory.

In another embodiment, although not shown, the processor may determine emotional information based on at least one object selected by user input. The processor may generate at least one graphic object corresponding to the determined emotional information and synthesize the at least one generated graphic object into an image.

In another embodiment, although not shown, when there are a plurality of objects selected by user input, the processor may determine a plurality of emotional information based on the plurality of objects. The processor may provide at least one graphic object to be synthesized into an image generated based on each of the determined plurality of emotional information. For example, the processor may provide at least one graphic object generated based on first emotional information and at least one graphic object generated based on second emotional information. The processor may synthesize the selected graphic object in the image in response to detecting an input for selecting one of at least one graphic object created based on each provided emotional information.

In another embodiment, although not shown, the processor may segment an image into a specific number of areas and recognize one or more objects included in the segmented areas. The processor may determine emotional information based on each of the one or more recognized objects. The processor may provide at least one graphic object to be synthesized in the divided area based on each of the determined emotional information. In response to detecting an input for selecting one of at least one graphic object created based on each emotional information for each provided region, the processor may synthesize the graphic object selected for the corresponding region.

In another embodiment, although not shown, the processor may determine emotional information based on each of one or more objects recognized in the image and then receive a plurality of voice data. When a plurality of voice data is received, the processor may segment the plurality of voice data. For example, the processor may segment the received plurality of voice data based on speech rate, acoustic tone, pitch, frequency, and/or sound pressure for each of the received plurality of voice data. The processor may determine emotional information for each of the plurality of voice data based on at least one of speech content, sound tone, pitch, frequency, or sound pressure of each of the plurality of received voice data. The processor may provide at least one graphic object based on emotional information for each of the determined voice data. The processor may generate at least one graphic object corresponding to emotional information for each of the determined voice data. The processor may determine at least one object to be synthesized based on the emotional information of each of the one or more objects, and may synthesize at least one object determined in the corresponding feature area of each of the one or more objects. In another embodiment, when there are a plurality of at least one graphic objects created, a user input for selecting at least one graphic object among the plurality of graphic objects may be detected. In response to a user input, the processor may synthesize at least one selected graphic object into a feature area of each of the one or more recognized objects.

FIG. 5 is a diagram 500 illustrating a method of determining emotional information by extracting a feature area for at least one object recognized from an image, according to various embodiments.

Referring to FIG. 5, at least one feature region ( Characteristics of (e.g., eye area, nose area, and/or mouth area) may be stored in memory (e.g., memory 230 of FIG. 2). For example, the characteristic of at least one feature area may include at least one of the shape of the eye (e.g., degree of drooping of the eye, focus of the eye, size of the eye, wrinkle of the eye) or shape of the mouth. A processor (eg, processor 270 of FIG. 2) may determine emotional information for each of at least one object based on the characteristics of at least one extracted feature area. For example, emotional information may include sadness (510), anger (520), surprise (530), and/or happiness (540). Although not limited to this and not shown, the emotional information may include worry, boredom, and/or disgust.

FIG. 6 is a diagram 600 illustrating a method of generating at least one graphic object based on determined emotional information, according to various embodiments.

Referring to FIG. 6, graphic objects and their properties generated based on emotional information may be stored in memory (eg, memory 230 of FIG. 2). For example, the processor (e.g., the processor 270 of FIG. 2) may process a graphic object based on emotional information stored in the memory and emotional information determined based on the properties of the graphic object, such as happy 610, surprised 610, etc. 620), at least one of font style 650, color 660, motion 670, or sticker 680 corresponding to emotional information such as angry 630, or sad 640, respectively. I can provide one.

FIG. 7 is a diagram 700 illustrating a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.

Referring to FIG. 7, the processor recognizes the face 713 of the object 711 included in the image as shown by reference numerals <710> and <720>, and detects at least one feature from the recognized face 713. Areas, such as the eye area 721 and the mouth area 723, can be extracted. The processor may determine emotional information about the recognized object 711 based on the extracted eye area 721 and mouth area 723. For example, the processor may determine the emotional information to be, for example, “angry” based on the extracted shapes of the eye area 721 and the mouth area 723.

In one embodiment, the processor may analyze the background 731 of the image (eg, an area excluding the recognized object 711 within the image) as shown by reference numeral <730>. The processor may analyze the background 731 of the image and determine the atmosphere of the image based on this. For example, the mood for the image can be determined based on the color (or saturation, brightness) of the background 731. However, it is not limited to this, and the processor processes the types of objects included in the image (e.g., weapons, objects such as skeletons, objects such as animals, dolls), the movement of the objects (e.g., fast movement, slow movement), and the frequency of the image. The mood for the image may be determined based on the components (e.g., high-frequency components of edges and boundaries, low-frequency components), and/or text included in the image. For example, if an object such as a weapon or skull is recognized, the movement of the object exceeds a specified speed, the frequency component of the image is a high-frequency component, or the text included in the image is text indicating negative emotion, the processor collects emotional information. You can decide between “fear” or “anger.” If an object such as an animal or doll is recognized, the movement of the object is below a specified speed, the frequency component of the image is a low-frequency component, or the text included in the image is text indicating a positive emotion, the processor classifies the emotional information as “happy.” ”, you can decide on “fun”.

In FIG. 7 according to one embodiment, the atmosphere determined based on the image will be described assuming that it is a creepy and scary atmosphere.

In one embodiment, the processor recognizes an object 711 from an image, at least one feature region 721, 723 for the object 711, determined emotional information (e.g., angry), and an atmosphere (e.g., eerie and scary). atmosphere) can be stored in memory (e.g., memory 230 in FIG. 2).

In one embodiment, the processor generates at least one graphical object based on the emotion information (e.g., angry) and the determined atmosphere (e.g., eerie and scary atmosphere) based on the extracted shapes of the eyes 721 and mouth 723. can be created.

In one embodiment, the processor may receive voice data, such as “watch out” 735, through a microphone. The processor may analyze the content of the received voice data, such as “watch out” 735, and the characteristics of the user's voice included in the voice data (e.g., tone, pitch, frequency, and/or sound pressure).

In one embodiment, the processor determines emotional information for the input voice data based on the content of the analyzed voice data and the characteristics of the user voice, as shown by reference numeral <740>, and creates at least one graphic based on this. An object 741 can be created. Based on the results of analyzing voice data, emotional information may be determined as, for example, “angry.”

In one embodiment, the processor may display at least one graphic object 751 to which attributes corresponding to the generated emotional information, as shown by reference numeral <750>, are applied. Attributes may include at least one of font size, style, color, thickness, or motion. For example, at least one graphic object to which an attribute is applied may include text generated based on voice data, such as “watch out” 735, and a sticker generated based on emotional information and mood.

In one embodiment, the processor may display a confirmation button 753 along with at least one graphic object 751 to which attributes corresponding to emotional information are applied. When a user input for selecting the confirm button 753 is detected, the processor generates at least one graphic object to which an attribute is applied, as shown by reference number <760>, for example, based on emotional information (e.g., angry) about voice data. Generated text 751 (e.g., text with a sharp-edged font and red color) and stickers 761 generated based on emotional information (e.g., angry) and mood (e.g., spooky and scary), e.g. Stickers with angry eyes) can be composited into the video. In one embodiment, the text 751 and the sticker 761 may each be synthesized by being placed in a corresponding feature region, such as the mouth region 723 and the eye region 721, among at least one extracted feature region.

In the above-described embodiment, in response to a user input of selecting the confirm button 753, as shown by reference number <750>, an image synthesized by at least one graphic object to which the attribute indicated by reference number <760> is applied is displayed. Although it has been explained as such, it is not limited to this. For example, after displaying at least one graphic object 751 to which attributes corresponding to the generated emotional information of reference number <750> are applied, if a specified time (e.g., 3 seconds) is exceeded, the processor displays the graphic object 751 of reference number <760>. An image synthesized from at least one graphic object to which properties are applied may be displayed.

In one embodiment, although not shown, the processor may display at least one graphic object 751 to which attributes corresponding to emotional information are applied and a button for customizing the graphic object 751. At least one graphic object corresponding to the emotional information intended by the user can be accurately created through a user input of selecting a button for customization.

FIG. 8 is a diagram 800 illustrating a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.

Referring to FIG. 8, the processor recognizes the face of an object included in an image as shown by reference numeral <810>, and selects at least one feature area, such as an eye area 811 and a nose area 813, from the recognized face. ), and/or the mouth area 815 can be extracted.

In one embodiment, the processor may receive voice data from a user, such as “smells delicious” (817). The processor, as shown by reference numeral <820>, generates at least one graphic, such as attributes (e.g., font size, style, color, thickness, and/or A first graphic object 821, a second graphic object 823, and a third graphic object 825 to which motions (motions) are applied differently may be displayed on the display.

In one embodiment, the processor detects a user input 827 to select a first graphical object 821 among the displayed first graphical object 821, a second graphical object 823, and a third graphical object 825. can do. In response to detecting the user input 827 for selecting the first graphic object 821, the processor may determine a position in the image to place the selected first graphic object 821, as shown by reference numeral <830>. there is. For example, the processor may analyze the speech content of the voice data and determine where to place the selected first graphic object 821 in the image. For example, the processor analyzes speech data “smells delicious” 817 such that “smells” is placed 831 in a feature area, such as the nose area 813, and “delicious” is placed 831 in a feature area, such as the mouth area 815. It may be decided to be placed 833.

It is not limited to this, and in another embodiment, the processor may receive an input from the user for setting a position in the image where the selected graphic object is to be placed.

In one embodiment, the processor may receive inputs 841 and 843 for setting the motion of the first graphic object 821, such as “smells” and “delicious,” as shown by reference numeral <840>. , Based on the inputs 841 and 843, the motion of each of the first graphic objects 821, such as “smells” and “delicious,” can be set. The processor creates an image of the first graphic object 821 based on the motions 851 and 853 set to the first graphic object 821, for example, “smells” and “delicious”, respectively, as shown in reference numeral <850>. can be synthesized.

FIGS. 9A and 9B are diagrams 900 for explaining a method of synthesizing at least one graphic object created at a position corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments. )am.

Referring to FIG. 9A, a processor (e.g., processor 270 of FIG. 2) processes the first face 913 and the second object of the first object 911 included in the image, as shown by reference numeral <910>. The second face 917 of 915 can be recognized. As shown in reference numeral <920>, the processor selects at least one feature area, such as an eye area, a nose area, And/or the mouth area may be extracted, and emotional information about the second object 915 may be determined based on the extracted feature area. For example, the emotional information determined based on the extracted feature area is explained by assuming that it is “angry.”

In one embodiment, the processor may receive first voice data, such as “what are you doing” 925. As shown in reference numeral <930>, the processor generates at least one message for the second face 917 of the second object 915 based on the received first voice data, for example, “what are you doing” 925. A graphic object 931 can be created and displayed. For example, the at least one graphic object may include text for received voice data. The processor may determine emotional information about the first voice data based on at least one of the speech content, sound tone, pitch, frequency, and sound pressure of the received first voice data. For example, the emotional information for the first voice data is explained by assuming that it is “tension” and “angry.” Determine an attribute (e.g., at least one of font size, style, color, thickness, or motion) of the text for the first voice data based on emotional information (e.g., nervous, angry) for the first voice data, and determine the determined The text “what are you doing” (931) to which the attribute is applied can be displayed.

In one embodiment, the processor generates second voice data, such as “nothing…” ”(935). For example, the second voice data may be received after the first voice data is received and a specified time has elapsed. The processor may determine emotional information about the second voice data based on at least one of the speech content, sound tone, pitch, frequency, and sound pressure of the received second voice data. For example, the emotional information for the second voice data is explained by assuming that it is “worried.” As shown in reference numeral <940>, the processor determines text attributes for the text for the second voice data based on emotional information (e.g., worry) for the second voice data, and determines the text attribute to which the determined attribute is applied. nothing… ”(941) can be displayed.

In one embodiment, the processor generates at least one graphic object (e.g., text and/or sticker) among at least one feature area based on the speech content of the voice data and the determined emotional information, as shown by reference numeral <950>. ) can be determined on the area within the image to be synthesized. The processor may display at least one graphic object generated in the determined area (e.g., text “what are you doing” (931) for the first voice data, text “nothing…” (941) for the second voice data, and/or A sticker 951 corresponding to emotional information determined based on the feature area of the second object 915 may be synthesized.

In FIG. 9A according to one embodiment, it has been described that voice data, such as first voice data and second voice data, are received from the user at designated time intervals, but the present invention is not limited thereto. For example, the processor may simultaneously receive voice data from each of a plurality of users.

The embodiment of simultaneously receiving voice data from each of the above-described plurality of users will be described in detail in FIG. 9B described later.

Referring to FIG. 9B, the processor processes the first face 963 of the first object 961 and the second face 967 of the second object 965 included in the image, as shown by reference number <960>. It can be recognized. The processor extracts at least one feature region (e.g., eye region, nose region, and/or mouth region) from the recognized first face 963 and second face 967 as shown in reference numeral <970>. After determining the emotional information for the first object 961 and the second object 965 based on , a plurality of voice data can be received from a plurality of users. For example, the processor receives first voice data “what are you doing” 973 from a plurality of users, such as a first user, and second voice data “nothing…” from a second user. ”(977). The processor may determine emotional information about the first voice data and the second voice data based on at least one of speech content, sound tone, pitch, frequency, and sound pressure of the received first voice data and the second voice data.

In one embodiment, the processor generates first voice data, such as “what are you doing” (973) and second voice data, such as “nothing…” ”(977), at least one graphic object can be created and displayed. For example, the at least one graphic object is text for the first voice data and the second voice data to which text properties (e.g., at least one of font size, style, color, thickness, or motion) determined based on emotional information are applied. may include.

In one embodiment, the processor may compare emotional information about the first object 961 and the second object 965 with emotional information about the first voice data and the second voice data. For example, the operation of comparing emotional information may be an operation of analyzing the similarity of the emotional information and/or whether the emotional information is of the same category (eg, positive emotional information, negative emotional information). Based on the comparison result, the processor may determine an area within the image in which to place at least one graphic object created based on the first voice data and the second voice data. For example, the emotional information for the first object 961 is “sad,” the emotional information for the second object 965 is “angry,” the emotional information for the first voice data is “angry,” and the emotional information for the first voice data is “angry.” Assuming that the emotional information for is “worry”, the processor determines the text for the second voice data having the emotional information “worry” as the area in which to place the feature area of the first object 961 having the emotional information “sadness”. And, the text for the first voice data having the emotional information “angry” can be determined as an area in which to place the feature area of the second object 965 having the emotional information “angry”.

In one embodiment, the processor generates at least one graphic object in the determined area as shown by reference numeral <980> (e.g., text “what are you doing” (981) for the first voice data, and the second voice data. The text “nothing…” (983) for the data and/or the sticker (985) corresponding to the emotional information determined based on the feature area for the second object (965) may be synthesized and displayed.

FIG. 10 is a diagram 1000 illustrating a method of synthesizing at least one graphic object created at a location corresponding to a feature area of at least one object based on determined emotional information, according to various embodiments.

Referring to FIG. 10, the processor processes a plurality of objects included in the image, for example, the first face 1013 of the first object 1011 and the first face 1013 of the second object 1015, as shown by reference number <1010>. 2 Faces (1017) can be recognized.

In one embodiment, the processor may receive voice data, such as “welcome back” (1019). As shown by reference numeral <1020>, the processor may generate at least one graphic object based on received voice data, such as “welcome back” (1019), and display it on the display (1021). For example, the at least one graphic object may include text for received voice data. The properties of text for voice data may be determined based on at least one of speech content, sound tone, pitch, frequency, and sound pressure of the received voice data. For example, emotional information determined based on voice data is explained assuming that “happy.”

In one embodiment, the processor may separate text from voice data based on the number of recognized objects, as shown by reference numeral <1030>. For example, text can be broken down into letters that make up words. For example, the graphic object 1031 to be placed on the first face 1013 of the first object 1011 is “welcome”, and the graphic object 1033 to be placed on the second face 1017 of the second object 1015 is “welcome”. can be decided as “back”.

In one embodiment, the processor generates at least one graphic object, such as stickers 1041, 103, and 1045, corresponding to emotional information, such as "happiness," determined based on voice data, as shown by reference numeral <1040>. It can be combined with the first face 1013 of the 1 object 1011 and the second face 1017 of the second object 1015.

In one embodiment, the processor displays each of the graphic object 1031 “welcome” and the graphic object 1033 “back” together with the stickers 1041, 1043, and 1045 as shown in reference numeral <1050> to a corresponding object, such as, It can be combined with the first face 1013 of the first object 1011 and the second face 1017 of the second object 1015.

FIG. 11 is a diagram 1100 illustrating signal flow between electronic devices that provide corresponding graphic objects corresponding to emotional information, according to various embodiments.

In the embodiment according to FIG. 11, the performance of the external electronic device 1120 may be higher than that of the electronic device 1110, and the high-performance external electronic device 1120 analyzes information related to the image and provides the analyzed result. can be provided to the electronic device 1110.

Referring to FIG. 11 , the electronic device 1110 may display an image in operation 1130. In one embodiment, the image may include at least one image captured from at least one camera (e.g., camera 220 of FIG. 2) and an image stored in a memory (e.g., memory 230 of FIG. 2). .

In one embodiment, the electronic device 1110 may determine information related to the image to be analyzed in operation 1135. For example, information related to the image to be analyzed may include a second image different from the image displayed in operation 1130, audio information included in the video, text information included in the video, biometric information of the user of the electronic device 1110, or situation information. It may include at least one of:

For example, the second image may be an image including the image displayed in operation 1130 (e.g., one of the frame images of the image, an image including at least part of the face, body, and object in the first image), a barcode, and a two-dimensional barcode ( (e.g. QR code, data matrix code), marker, or text image may be included. Audio information may include audio track information of voice, sound signals, and/or images received by the electronic device 1110. Biometric information is biometric information received through the biometric sensor of the electronic device 1110 (e.g., HRM, HRV, SPO2, blood pressure, blood sugar, EEG, skin resistance, electromyogram, electrocardiogram, respiratory rate per minute, or body temperature), electronic device It may include biometric information (e.g., blood flow change image) obtained by analyzing the image through the camera at 1110, and/or body pose or body movement information using a motion sensor. Text information may include text, symbols, signs, video caption tracks, associated caption file information, and subtitle information included in or associated with music content by pen input, virtual keyboard, or keypad. Contextual information includes the current time, current location, schedule-related information (e.g., birthday, Christmas) for the electronic device 1110, and profile information (e.g., preferences, food, personal information) about the user of the electronic device 1110. (e.g. gender, age, visit information)). In one embodiment, the emotion analysis target information may be included in the video or may be received separately (e.g., voice data).

In one embodiment, the electronic device 1110 may transmit information related to the image to be analyzed to the external electronic device 1120 in operation 1140.

In one embodiment, the external electronic device 1120 may analyze information related to the image received from the electronic device 1110 and determine emotional information based on this in operation 1145. For example, the external electronic device 1120 is based on at least one of a second image different from the displayed image, audio information included in the video, text information included in the video, biometric information of the user of the electronic device 1110, or situation information. Thus, emotional information can be determined.

In one embodiment, the external electronic device 1120 may determine at least one graphic object based on the determined emotional information in operation 1150. In another embodiment, at least one graphic object may be determined based on the preferences of the user of the electronic device 1110 or another user.

In one embodiment, the external electronic device 1120 may transmit the determined emotional information and at least one graphic object to the electronic device 1110 in operation 1155. In operation 1160, the electronic device 1110 may determine a position to composite at least one graphic object in the image. In operation 1165, the electronic device 1110 may determine a graphic object to be composited into an image among at least one received graphic object, and in operation 1170, composite the determined graphic object at the determined position.

FIG. 12 is a diagram 1200 illustrating signal flow between electronic devices that provide corresponding graphic objects corresponding to emotional information, according to various embodiments.

In the embodiment according to FIG. 12, the first external electronic device 1210 receives analysis target information and provides the received analysis target information to the electronic device 1220 (e.g., electronic device 1110 in FIG. 11). You can. For example, the first external electronic device 1210 may include a smart watch, a head mounted display (HMD), an augmented reality glass (AR glass), a drone, a robot, or an Internet of Things (IoT) device. In one embodiment, the first external electronic device 1210 can recognize and process video or audio information, and can also receive video and transmit it to the electronic device 1220.

In one embodiment, the performance of the second external electronic device 1230 (e.g., the first external electronic device 1120 in FIG. 11) may be higher than that of the electronic device 1220 and the first external electronic device 1210. Information related to the image can be analyzed in the high-performance second external electronic device 1230, and the analyzed results can be provided to the electronic device 1220. For example, information related to an image may include video, audio, sound-to-text (STT), barcode, marker, and emotional information (or emotion, atmosphere). In one embodiment, the second external electronic device 1230 may manage preference information based on user profiles received from electronic devices and provide graphic objects based on the preference information. The second external electronic device 1230 can store and manage graphic objects and transmit them to the electronic device 1220. When a request is received from the electronic device 1220, the determined candidate graphic object and image can be received, and synthesis and editing operations can be performed. For example, the second external electronic device 1230 may be a server, a smart home hub device, a cloud computing device, a mobile edge comp. (MEC), an access point device, an internet message server (IMS), and a multimedia streaming device. It may include either a server or a video communication host device.

Referring to FIG. 12 , the first external electronic device 1210 may obtain situation information about the first external electronic device 1210 in operation 1241. For example, the context information may include image information, audio information, text data, and/or biometric information acquired from the first external electronic device 1210.

In one embodiment, the electronic device 1220 may display an image in operation 1243.

In one embodiment, the first external electronic device 1210 may transmit the obtained context information to the electronic device 1220 in operation 1245.

In one embodiment, the electronic device 1220 may analyze at least one of at least one feature area extracted from the image, data related to the image, and context information obtained from the first external electronic device 1210 in operation 1247. there is. The electronic device 1220 may transmit the analysis result to the second external electronic device 1230 in operation 1249.

In one embodiment, the second external electronic device 1230 may determine emotional information based on the received analysis result in operation 1251.

The electronic device 1220 according to an embodiment analyzes at least one of at least one feature area extracted from the image, data related to the image, and situation information obtained from the first external electronic device 1210, and then provides the analysis result. Although it has been described as transmitting to the second external electronic device 1230, it is not limited to this. The electronic device 1220 determines at least one of at least one feature area extracted from the image, data related to the image, and situation information obtained from the first external electronic device 1210 as information related to the image to be analyzed, and selects it as information related to the image to be analyzed. 2 Can be transmitted to an external electronic device 1230. The second external electronic device 1230 may analyze information related to the image received from the electronic device 1220 and determine emotional information based on this.

In one embodiment, the second external electronic device 1230 may transmit the determined emotional information to the electronic device 1220 in operation 1253.

In one embodiment, the electronic device 1220 may determine a position to display a graphic object in an image in operation 1255. In operation 1257, the electronic device 1220 may request a graphic object to be composited into an image from the second external electronic device 1230.

In one embodiment, the second external electronic device 1230 determines at least one graphic object based on the determined emotional information in operation 1259, and sends the determined at least one graphic object to the electronic device 1220 in operation 1261. Can be transmitted.

In one embodiment, the electronic device 1220 may determine a graphic object to be composited into an image among at least one graphic object received from the second external electronic device 1230 in operation 1263. In operation 1265, the electronic device 1220 may request the second external electronic device 1230 to synthesize a graphic object determined in the image.

In one embodiment, the second external electronic device 1230 may synthesize a graphic object determined at a determined position in the image in operation 1267 and transmit the image in which the graphic object is synthesized to the electronic device 1220 in operation 1269. .

In one embodiment, the electronic device 1220 may receive and display an image in which a graphic object is synthesized from the second external electronic device 1230 in operation 1271.

According to the above-described embodiments of FIGS. 3 to 12, the processor selects at least one feature area extracted for at least one object recognized in the image, data related to the image, situation information, or information related to the analyzed voice data. Emotion information may be determined based on at least one emotion information, and at least one graphic object may be generated based on the determined emotion information. The processor may synthesize at least one object created based on emotional information into an image and store it in memory to enable communication with other users. As text to which attributes corresponding to at least one feature area, image-related data, and situational information as well as emotional information related to received voice data are applied are synthesized, the user's emotions can be conveyed more richly to other users.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smart phones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. In this document, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B, or C”, “at least one of A, B, and C”, and Each of phrases such as “at least one of A, B, or C” may include any one of the items listed together in that phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one element from another, and may be used to distinguish such elements in other respects, such as importance or order) is not limited. One (e.g. first) component is said to be “coupled” or “connected” to another (e.g. second) component, with or without the terms “functionally” or “communicatively”. Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single entity or a plurality of entities. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

210: wireless communication circuit 220: camera
230: memory 240: sensor unit
250: audio processing circuit 260: touch screen display
270: processor

Claims (20)

In electronic devices,
display;
mike;
a processor operatively connected to the display and the microphone; and
comprising a memory operatively coupled to the processor,
The memory, when executed, the processor,
Displaying an image on the display,
Recognize at least one object from the displayed image,
Extracting at least one feature area for the at least one recognized object,
Based on the extracted at least one feature region, determine first emotional information,
When at least one voice data is received through the microphone, determine second emotion information based on the at least one received voice data,
Generate a first graphic object corresponding to at least one of the first emotional information and the second emotional information,
Generating a second graphic object related to the at least one voice data to which attributes corresponding to the second emotional information are applied,
Determining a feature region to combine the first graphic object and the second graphic object among the at least one extracted feature region, and
Storing instructions for combining the first graphic object and the second graphic object at a location corresponding to the determined feature area,
The first graphic object includes at least one of a sticker and an emoji corresponding to at least one of the first emotional information and the second emotional information, and
The second graphic object is an electronic device including text related to the at least one voice data to which at least one of font size, style, color, thickness, or motion corresponding to the second emotional information is applied. According to claim 1,
The first emotional information is determined based on at least one of data related to the image or situational information,
Data related to the image includes at least one of a background of the image, a type of object placed in the foreground of the image, a color of the image, audio included in the image, or text, and
The situation information includes at least one of current time, current location, or event schedule,
The instructions are such that the processor,
displaying at least one first graphic object corresponding to first emotional information on the display, and
An electronic device configured to synthesize the selected graphic object at a position corresponding to the determined feature area, based on a user input for selecting one of the at least one graphic object corresponding to the displayed first emotional information. According to claim 2,
The instructions are such that the processor,
Determine third emotion information based on the first emotion information and the second emotion information, and
An electronic device that generates a third graphic object related to the third emotion information based on the determined third emotion information. delete According to claim 2,
The instructions are such that the processor,
When a priority is set for the at least one feature area, the data related to the image, or the context information, based on the information with a higher priority among the at least one feature area, the data related to the image, or the context information An electronic device that determines the first emotional information. According to claim 1,
The instructions are such that the processor,
When at least one voice data is received through the microphone, the speech content for each of the at least one received voice data, the tone, pitch, frequency, or sound pressure of the user's voice included in each of the at least one received voice data. An electronic device that analyzes at least one of the following: delete According to claim 1,
The instructions are such that the processor,
If there are a plurality of texts to which the attributes are applied, displaying a plurality of texts to which the plurality of attributes are applied on the display, and
An electronic device that, upon detecting a user input for selecting one of the plurality of texts to which the displayed plurality of attributes are applied, synthesizes the selected text at a position corresponding to the determined feature area. According to claim 1,
The instructions are such that the processor,
An electronic device that stores an image composed of the first graphic object and the second graphic object in the memory. delete In electronic devices,
display;
a processor operatively connected to the display; and
comprising a memory operatively coupled to the processor,
The memory, when executed, the processor,
Displaying an image on the display,
Recognize at least one object from the displayed image,
Determine emotional information based on at least one of data related to the image, situation information, or information related to a user of the electronic device,
Detecting a first user input for selecting a specific area among a plurality of areas of the image,
Based on the characteristics of the specific area where the first user input was detected, displaying one or more graphic objects corresponding to the determined emotional information to be synthesized in the specific area on the display, and
When a second user input for selecting at least one graphic object among the displayed one or more graphic objects is detected, instructions for synthesizing the selected at least one graphic object at a location corresponding to the specific area are stored. do,
The specific area includes at least one of a feature area of the object, an area where an animation character object exists, and an area where a background image is displayed, and
The one or more displayed graphic objects are different depending on the characteristics of the specific area where the first user input is detected. delete delete In a method of providing a graphic object corresponding to emotional information of an electronic device,
An operation of displaying an image on a display;
Recognizing at least one object from the displayed image;
extracting at least one feature area for the at least one recognized object;
An operation of determining first emotional information based on the extracted at least one feature area;
When at least one voice data is received through a microphone, determining second emotion information based on the at least one received voice data;
Generating a first graphic object corresponding to at least one of the first emotional information and the second emotional information;
generating a second graphic object related to the at least one voice data to which attributes corresponding to the second emotional information are applied;
determining a feature region to be combined with the first graphic object and the second graphic object among the at least one extracted feature region; and
Comprising the operation of combining the first graphic object and the second graphic object at a position corresponding to the determined feature area,
The first graphic object includes at least one of a sticker and an emoji corresponding to at least one of the first emotional information and the second emotional information, and
The second graphic object includes text related to the at least one voice data to which at least one of font size, style, color, thickness, or motion corresponding to the second emotional information is applied. delete delete delete delete In a method of providing a graphic object corresponding to emotional information of an electronic device,
An operation of displaying an image on a display;
Recognizing at least one object from the displayed image;
determining emotional information based on at least one of data related to the image, situational information, or information related to a user of the electronic device;
detecting a first user input for selecting a specific area among a plurality of areas of the image;
displaying on the display one or more graphic objects corresponding to the determined emotional information to be synthesized in the specific area based on characteristics of the specific area in which the first user input was detected; and
When a second user input for selecting at least one graphic object among the displayed one or more graphic objects is detected, compositing the selected at least one graphic object at a position corresponding to the specific area;
The specific area includes at least one of a feature area of the object, an area where an animation character object exists, and an area where a background image is displayed, and
The one or more graphic objects displayed are different depending on characteristics of a specific area where the first user input is detected. delete

Images