KR20220138669A - Electronic device and method for providing personalized audio information - Google Patents

Abstract

The present invention relates to an electronic device for providing personalized audio information, an operation method for an electronic device, and a non-transitory storage medium. In accordance with one embodiment, the electronic device includes: an audio module; a memory; and at least one processor electrically connected with the audio module and the memory. The at least one processor can be set to: receive user interaction information; identify characteristic information of a target object based on the received user interaction information; analyze the identified characteristic information to identify at least one speaker; generate personalized audio information corresponding to the characteristic information based on the identified at least one speaker; map the personalized audio information with the target object to generate personalized content; and store and execute the personalized content. Besides, other embodiments can be possible. Therefore, the present invention is capable of improving user experience by providing voices from a plurality of speakers.

Description

ELECTRONIC DEVICE AND METHOD FOR PROVIDING PERSONALIZED AUDIO INFORMATION

Various embodiments of the present document relate to an electronic device and method for providing personalized audio information.

Recently, electronic devices have been developed in various forms for the convenience of users, and various services or functions are provided.

The user's personal information may be included among the information according to the execution of various services or functions of the electronic device.

Conventionally, content is directly produced and shared in various ways according to various services or functions. For example, a video before a wedding ceremony, content to be uploaded from a sharing service, a video of a birthday party, a photo, a ppt slide video, etc. are produced and shared. Generally, music is used as a background for produced content, but narration is used for special content.

When providing a voice such as a narration to the content provided in the conventional content creation and sharing service, a customized voice is applied through various channels, or audio provided by a content manufacturer, such as a simple audio effect sound or the voice of a celebrity. Since voice is applied only with information, it was difficult to provide personalized voice suitable for content.

According to an embodiment of the present document, an electronic device and method for providing personalized audio information on content to which a personalized voice is applied may be provided.

According to an embodiment of the present document, an electronic device includes an audio module, a memory, and at least one processor electrically connected to the audio module and the memory, wherein the at least one processor receives user interaction information, Identifies characteristic information of a target object based on the received user interaction information, identifies at least one speaker by analyzing the identified characteristic information, and personalization corresponding to the characteristic information based on the identified at least one speaker It may be configured to generate audio information, create personalized content by mapping the personalized audio information to the target object, and store and execute the personalized content.

According to an embodiment, the method of operation in the electronic device includes receiving user interaction information, identifying characteristic information of a target object based on the received user interaction information, and analyzing the identified characteristic information to obtain at least one identifying a speaker of , generating personalized audio information corresponding to the feature information based on the identified at least one speaker, mapping the personalized audio information to the target object to generate personalized content; and It may include the operation of storing and executing personalized content.

According to an embodiment, in a non-transitory storage medium storing a program, when the program is executed by a processor, the processor receives user interaction information; Identifying characteristic information of a target object based on the received user interaction information, analyzing the identified characteristic information to identify at least one speaker, and corresponding to the characteristic information based on the identified at least one speaker and instructions executable to execute an operation of generating the personalized audio information, an operation of generating the personalized content by mapping the personalized audio information to the target object, and an operation of storing and executing the personalized content.

According to an embodiment, by providing an electronic device and method for providing personalized audio information, personalized audio information may be mapped to input content to generate and provide content to which a personalized voice is applied, and thereby, audio according to the content It can provide information and can provide the voices of various speakers, so the user experience can be improved, and the received content information can be easily checked during other tasks. In addition, various effects may be provided.

1 is a block diagram of an electronic device in a network environment, according to various embodiments of the present disclosure;
2 is a diagram illustrating a configuration of an electronic device for wireless communication according to an embodiment.
3 is a diagram illustrating a configuration of an electronic device for wireless communication according to an embodiment.
4 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
5 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
6 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
7A is a diagram illustrating an example of a database for an operating method in an electronic device according to an embodiment.
7A is a diagram illustrating an example of a speaker voice audio signal characteristic spectrum file for an operating method in an electronic device according to an embodiment.
8A is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
8B is a diagram illustrating an example of a database for an operating method in an electronic device according to an embodiment.
9 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
10 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
11 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
12A to 12E are diagrams illustrating an example of speaker selection in an operating method in an electronic device according to an exemplary embodiment.
13 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
14A is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
14B is a diagram illustrating an example of a database for an operating method in an electronic device according to an embodiment.
15 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
16 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
17 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
18 is a diagram illustrating an example of a database for an operating method in an electronic device according to an embodiment.
19 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

Hereinafter, an electronic device according to various embodiments will be described with reference to the accompanying drawings. The term user used in various embodiments may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with at least one of the electronic device 104 and the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 . In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in , process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 uses less power than the main processor 121 or is set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The secondary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190 ). have. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . The electronic device 102) (eg, a speaker or headphones) may output a sound.

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various techniques for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements defined in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to one embodiment, the wireless communication module 192 is a peak data rate (eg, 20 Gbps or more) for realizing 1eMBB, a loss range (coverage) for realizing mMTC (eg, 164 dB or less), or URLLC U-plane latency (eg, downlink (DL) and uplink (UL) respectively 0.5ms or less, or round trip 1ms or less) for realization may be supported.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. The server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a diagram illustrating a configuration of an electronic device according to various embodiments of the present disclosure, and FIG. 3 is a diagram illustrating a configuration of an electronic device according to various embodiments of the present disclosure.

1 and 2 , an electronic device 101 according to an embodiment may implement a software module 201 (eg, the program 140 of FIG. 1 ) for providing personalized audio information. The memory 130 of the electronic device 101 may store instructions (eg, instructions) to implement the software module 201 illustrated in FIG. 2 . The at least one processor 120 may execute instructions stored in the memory 130 to implement the software module 201 shown in FIG. 2 , and hardware (eg, FIG. 1 ) associated with the function of the software module 201 . of the sensor module 176, the power management module 188, or the communication module 190).

Referring to FIG. 2 , the software module 201 of the electronic device 101 according to an embodiment includes an application 210 (eg, the application 146 of FIG. 1 ) and a framework (eg, the middleware 144 of FIG. 1 ). )) 220 , and a kernel 230 (or HAL). At least a portion of the software module 201 may be preloaded on the electronic device 101 or may be downloadable from a server (eg, the server 108 ).

According to an embodiment, the application 210 may be set to include an application (eg, a module, a manager, or a program) related to location search. For example, the application 210 may include various applications (app1 , app2 , and app3 ) related to the provision of personalized audio information by the electronic device 101 . For example, the application 210 may include an application related to reproduction of an image, video, or audio, an application related to content creation, and/or an application related to content sharing.

The application 210 may include an application received from an external electronic device (eg, the server 108 or the electronic devices 102 and 104 ). According to an embodiment, the application 210 may include a preloaded application or a third party application downloadable from a server. Components of the software module 201 according to the illustrated embodiment and names of components may vary depending on the type of the operating system. According to an embodiment, at least a portion of the software module 201 may be implemented in software, firmware, hardware, or a combination of at least two or more thereof. At least a part of the software module 201 may be implemented (eg, executed) by, for example, a processor (eg, an AP). At least a portion of the software module 201 may include, for example, a module, a program, a routine, a set of instructions, or a process for performing at least one function.

According to an embodiment, the framework 220 provides a function commonly required by the applications 210 or enables the application 210 to efficiently use the limited system resources inside the electronic device 101 . Various functions may be provided to the application 210 through an application programming interface (API) (not shown). The framework 220 may include a personalized audio manager (PAM) 221 for providing personalized audio information, and a window manager interworking with the personalized audio management module 221 . ) 222 , a viewer system module 223 , a sensor manager 224 , an activity manager 225 and/or a system user interface module 227 ) may be included. The framework 220 is not limited thereto and may be configured to further include other modules. The framework 220 may include modules that form combinations of various functions of the above-described components. The framework 220 may provide a specialized module for each type of operating system in order to provide differentiated functions. The framework 220 may dynamically delete some existing components or add new components.

According to an embodiment, the HAL 230 may be configured to include an event hub 231 and a surface finger 233 . The HAL 230 may be included in the kernel 240 and configured. The kernel 240 may include a sensor driver 241 and a DDI driver 243 that interwork with a hardware module 250 (eg, a sensor controller) 251 and a display panel 253, It may include a system resource manager or a device driver, but is not limited thereto, and may be configured to further include other modules. The system resource manager may control, allocate, or reclaim system resources. The device driver may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WIFI driver, an audio driver, or an inter-process communication (IPC) driver.

According to an embodiment, the processor 120 of the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 (hereinafter, referred to as the first electronic device 101 )) is a software-configured framework 220 . It is possible to execute the instructions stored in the memory 130 to implement the personalized audio management module 221 included in the . For example, the personalized audio management module 221 may include at least the processor 120 implemented in software. It may be some component.

According to an embodiment, the processor 120 may receive user interaction information in response to execution of a related application capable of acquiring user interaction information. The processor 120 may identify the reception of the interaction information as a trigger action (or event) for initiating a function for mapping personalized audio. For example, the event may be an interface execution operation of executing instructions stored in the memory 130 to execute the personalized audio management module 221 when receiving at least one of audio, video, image, or text included in the interaction information. have.

According to an embodiment, the processor 120 may execute the personalized audio management module 221 in response to an event, and the personalized audio management module 221 may identify characteristic information of a target object. can The processor 120 generates personalized audio information (eg, TTS audio) based on the characteristic information identified by the personalized audio management module 221 and maps (or mixes) the personalized audio information generated to the target object. You can create content to run. The processor 120 may store and execute the generated content in the memory 130 . The target object may mean data such as a photo, GIF, audio, or video, and may be a part of data included in the received interaction information. The specific information may mean at least one data type (eg, mime type) of audio, video, image, and text. For example, audio information (or file) is an audio format supported by the electronic device (eg wav, mp4, ogg, gsm, dct, flac, au, aiff, vox, raw, aac, atrac, ra, ram). , dss, msv, dvf, etc.). In addition, when the audio information (or file (File)) is transmitted in the form of data to the network through a communication module (eg, the communication module 180 in FIG. 1 ) through an audio conversion device (Covert) (not shown) It may refer to a format stored in a memory (eg, the memory 130 of FIG. 1 ) corresponding to the user interface.

2 and 3 , the personalized audio management module 221 according to an embodiment includes a target object decision module (TODM) 310 and a target object analysis module (TOAM) 320 , a personalized content mapping module (PCMM) 330 and a personalized content save module (PCSM) 340 . The personalized audio management module 221 may include a database module 350 . Here, the database module 350 may be included in the electronic device 101 or configured outside the electronic device 101 separately from the personalized audio management module 221 . According to an embodiment, the target object determination module 310 is a module for determining the characteristic information of the target object for each data type (eg, mime type) of the characteristic information and determining the analysis operation of the characteristic information, the audio determination module ( a target object decision audio module (TODAM) (not shown), a target object decision video module (TODVM) (not shown), a target object decision image module (TODIM) (not shown), and It may include a target object decision text module (TODTM) (not shown). According to an embodiment, the target object analysis module 320 is a module for performing an analysis operation of the characteristic information determined by the target object determination module 310, and an audio analysis module (TOAAM: target object analysis audio module) 321 . , including a target object analysis video module (TOAVM) (322), a target object analysis image module (TOAIM) (323) and a target object analysis text module (TOATM) (324). can do. The present invention is not limited thereto, and the personalized audio management module 221 of the electronic device 101 may be configured by adding, replacing, changing, or excluding at least some components depending on the platform included in the electronic device 101 .

According to an embodiment, the audio analysis module 321 may perform an operation of analyzing the audio information by the audio analysis operation determined by the target object determination module 310 . The audio analysis module 321 acquires audio information as a target object by executing a related application (eg, an interview, a phone call, or a music application), separates a human voice from the acquired audio information, and analyzes the separated voice. Recognizes a speaker, converts speech to text (STT) for each section to generate a speaker identifier (ID), and uses the generated speaker identifier in a designated database (eg, speaker voice audio information DB) of the database module 350 . ) can be stored in Here, the speaker identifier (ID) is at least one of a personalized voice (eg, a girlfriend's voice, a family voice, a child's voice, one's own voice, a celebrity voice, or a voice extracted from audio) to be used when mapping personalized audio information to a target object. ) may be an identifier indicating the mapped speaker. According to another embodiment, at least some operations of the audio analysis module 321 may be performed by the audio determination module TODAM of the target object determination module 310 .

According to an embodiment, the video analysis module 322 may perform an operation of analyzing the video information by the video analysis operation determined by the target object determination module 310 . The video analysis module 322 obtains video information as a target object by executing a related application (eg, a video production or playback application), separates audio information and image information from the obtained video information, and separates the audio information from the audio information. A speaker is recognized by analyzing a voice, a speaker identifier (ID) is generated by converting speech to text (STT) for each section, and the generated speaker identifier is used in a designated database (eg, speaker) of the database module 350 . voice audio information DB). The video analysis module 322 may analyze the scene and generate the analyzed scene as text. According to another embodiment, at least some operations of the video analysis module 322 may be performed by the video determination module (TODVM) of the target object determination module 310 .

According to an embodiment, the image analysis module 323 may perform an operation of analyzing image information by an image analysis operation determined by the target object determination module 310 . The image analysis module 323 may acquire image information as a target object by executing a related application (eg, a camera, SNS, email, or Internet application), and may generate descriptive text by analyzing the acquired image information. According to another embodiment, at least some operations of the image analysis module 323 may be performed by the image determination module TODIM of the target object determination module 310 .

According to an embodiment, the text analysis module 324 may perform an operation of analyzing text information by the text analysis operation determined by the target object analysis module 310 . The text analysis module 324 may acquire text information as a target object by executing a related application (eg, SNS, email, or memo application), and correct typos and profanity in the acquired text information. For example, when a text-to-speech function (TTS) is executed in a situation in which it is difficult for the user to check the received text message (eg, while driving or doing other work), the text analysis module 324 . may transmit the text information to the personalized content mapping module 330 to map the personalized voice to the text information included in the received text message. According to another embodiment, at least some operations of the text analysis module 324 may be performed in the text determination module (TODTM) of the target object determination module 310 .

According to an embodiment, the personalized content mapping module 330 corresponds to the identified feature information and at least one speaker (or speaker identifier) stored in a designated DB (eg, speaker voice audio information DB) of the database module 350 can be mapped. The personalized content mapping module 330 may select at least one speaker (or speaker identifier) to be mapped through at least one of an explicit speaker selection, a quoted phrase speaker selection, or a speaker selection method by tone from a designated DB. For example, if the sender of the target object is explicit, the personalized content mapping module 330 may use an explicit speaker selection method. If the sender of the target object is not explicit, the personalized content mapping module 330 may use one of a quoted phrase speaker selection method or a speaker selection method by tone as an implicit speaker selection method.

According to an embodiment, the personalized content mapping module 330 searches for a contact identifier corresponding to the sender in the contact DB of the database module 350 using an explicit speaker selection method, and the target object DB of the database module 350 The identified speaker identifier may be selected by searching for a target object identifier corresponding to the searched contact identifier, and identifying the speaker identifier mapped with the contact identifier in the speaker voice audio information DB of the database module 350 . The personalized content mapping module 330 may generate mapping information in which the searched speaker identifier and the target object identifier are mapped, and store the generated mapping information in the speaker mapping table DB of the database module 350 .

According to an embodiment, if the sender is not explicit, the personalized content mapping module 330 uses a quoted phrase speaker selection method to correspond to a name included in data (eg, message) included in the received interaction information. Retrieve the descriptive text from the descriptive text DB of the database module 350, identify a target object identifier mapped to the retrieved descriptive text, identify a contact identifier mapped to the identified target object identifier in the target object DB, The identified speaker identifier may be selected by identifying the speaker identifier mapped to the contact identifier identified in the speaker voice audio information DB. The personalized content mapping module 330 may generate mapping information in which the searched speaker identifier and the target object identifier are mapped, and store the generated mapping information in the speaker mapping table DB.

According to an embodiment, the personalized content mapping module 330 uses a speaker selection method for each tone when the sender is not explicit or does not exist in the contact DB, data (eg, message) included in the received interaction information. Since there is no matching information, a tone analysis is performed to obtain an analysis result (eg, serious, serious, and official), the obtained analysis result is searched in the tone information DB of the database module 350, and the tone mapped to the searched tone identifier can be identified. The personalized content mapping module 330 may select the identified speaker identifier by identifying the speaker identifier mapped to the tone identifier identified in the speaker voice information DB. The personalized content mapping module 330 may generate mapping information in which the searched speaker identifier and the target object identifier are mapped, and store the generated mapping information in the speaker mapping table DB. The personalized content mapping module 330 may use a sentimental analysis model for tone analysis. The emotion analysis model can calculate a weight by generating a dictionary (corpus) from each word included in a text file, vectorizing the words (word2vec), and passing them through multiple layers of neurons. The personalized content mapping module 330 may acquire a tone by learning a received text message or a text file of an audio, image, or video through an emotion analysis model.

According to an embodiment, the personalized content mapping module 330 converts the descriptive text of the video object, the descriptive text of the image object, or the text of the text object into speech based on the audio characteristic value mapped to the selected speaker identifier (TTS). Personalized audio information that is TTS audio may be generated. When the target object is a moving picture including time information, the personalized content mapping module 330 may reprocess TTS audio tracks by mapping a plurality of TTS audios to a key video frame. The plurality of TTS audio files may be reprocessed into a TTS audio file whose duration is adjusted, and may be applied to an operation of mixing a target object and personalized audio information that is TTS audio.

According to an embodiment, the personalized content mapping module 330 may generate content to which the personalized audio information is applied as a product to be provided by mixing the target object and personalized audio information that is the generated TTS audio. For example, when the target object is a text object, the personalized content mapping module 330 may use the generated TTS audio as dynamic content. For example, when the target object is an image target object, the personalized content mapping module 330 arranges the image according to the TTS audio, and then arranges the image in each frame of the video in the video module (not shown) of the video encoder. dynamic content in the form of a moving picture by mapping the data, transmitting TTS audio to the audio module (not shown) of the video encoder, mixing TTS audio with the audio part of the moving picture, and encoding and synthesizing the arranged images and TTS audio by the encoder can create Here, the personalized content mapping module 330 may arrange the images based on the audio description for each region of the original image. For example, when the target object is a video object, the personalized content mapping module 330 may mix an audio part of the video object and TTS audio and encode and synthesize by an encoder to generate dynamic content in the form of a video. Here, since the encoder is a known technology, a detailed description of the encoding operation will be omitted.

According to an embodiment, the personalized content storage module 340 mixes the target object and the personalized audio information (TTS audio) by the personalized content mapping module 330 to store content (eg, dynamic content) generated by the memory 130 . can be stored in The personalized content storage module 340 may determine a stored type corresponding to a data type (eg, mime type) of specific information of the target object. The personalized content storage module 340 may execute the generated content (eg, dynamic content) through the audio module 170 and/or the display module 160 . For example, the content may be played by mapping a personalized voice to the generated video content. The personalized content storage module 340 may execute content according to an event of a user interaction in response to an interface displayed on the display module 160 . Here, the interface may mean a content execution application. The event may mean an operation of executing content. For example, the personalized content storage module 340 may generate and store content of an audio file in the case of an audio object, and may generate and store content of a video file in the case of a moving image or image object. For another example, when the network data transmitted from the network during real-time streaming is stored in the memory buffer, the personalized content storage module 340 may store and share the data in the buffer without storing it in a file.

As such, in an embodiment, the main components of the electronic device have been described with reference to the electronic device 101 of FIGS. 1 and 3 . However, in various embodiments, not all of the components illustrated in FIGS. 1 and 3 are essential components, and the electronic device 101 may be implemented by more components than the illustrated components, or fewer components. The electronic device 101 may be implemented by Also, positions of major components of the electronic device 101 described above with reference to FIGS. 1 and 3 may be changeable according to various embodiments.

According to an embodiment, the electronic device (eg, the electronic device 101 of FIGS. 1 and 3 ) includes an audio module (eg, the audio module 170 of FIG. 1 ) and a memory (eg, the memory 130 of FIG. 1 ). ) and at least one processor (eg, the processor 120 of FIG. 1 ) electrically connected to the audio module and the memory, wherein the at least one processor receives user interaction information and receives the received user interaction Identifies characteristic information of the target object based on the information, identifies at least one speaker by analyzing the identified characteristic information, and generates personalized audio information corresponding to the characteristic information based on the identified at least one speaker, and , create personalized content by mapping the personalized audio information to the target object, and store and execute the personalized content.

According to an embodiment, the target object represents at least one of audio, video, image, and text data included in the user interaction information, and the feature information includes at least one data type of audio, video, image, or text. can do.

According to an embodiment, when the feature information is identified as an audio type, the at least one processor analyzes the audio information of the target object to classify at least one user voice from the audio information, and Set to identify at least one speaker based on the tone of one user's voice, and generate the personalized audio information corresponding to the at least one speaker based on an audio signal characteristic value mapped to the identified at least one speaker can be

According to an embodiment, the at least one processor is configured to identify the at least one speaker using a speaker voice audio information database stored in the memory, and to identify an audio signal characteristic value mapped to the at least one speaker can be

According to an embodiment, when the feature information is identified as an audio type, the at least one processor analyzes the audio information of the target object to divide the audio information into sections, and the divided sub-audio for each section Converts information into text, generates descriptive text corresponding to the converted text for each section, and compares the descriptive texts generated for each section with mapping information stored in the target object database stored in the memory to select at least one speaker and to generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the identified at least one speaker.

According to an embodiment, when the feature information is identified as a video type, the at least one processor analyzes video information of the target object and extracts at least one image for at least one main scene from the video information and generating descriptive text corresponding to the scene of the at least one image based on a result of analyzing the scene of the at least one image, and using the generated descriptive text with mapping information stored in the target object database stored in the memory; The comparison may be configured to identify at least one speaker, and generate personalized audio information corresponding to the at least one speaker based on an audio signal characteristic value mapped to the identified at least one speaker.

According to an embodiment, when the feature information is identified as an image type, the at least one processor analyzes the image information of the target object to analyze at least one main scene from the image information, and collects the analysis result. generating descriptive text corresponding to the at least one main scene based on the at least one main scene, comparing the generated descriptive text with mapping information stored in a target object database stored in the memory to identify at least one speaker, and the identified at least one It may be configured to generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the speaker of .

According to an embodiment, when the characteristic information is identified as a text type, the at least one processor analyzes image information of the target object to identify a typo or profanity in the text information, and the identified typo or profanity Generates corrected text information by replacing It may be configured to generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the speaker.

According to an embodiment, when the user information is identified based on the analysis result of the target object, the at least one processor is mapped to the user information using a contact database and a speaker voice audio information database stored in the memory. It may be configured to identify the at least one speaker.

According to an embodiment, when the user information is not identified based on the analysis result of the target object, the at least one processor converts text information included in the target information or at least some data of the target information. and identify the at least one speaker mapped to the descriptive text using a descriptive text database, a target object database and a speaker voice audio information database stored in the memory.

4 is a diagram illustrating an example of an operation method in an electronic device according to an embodiment.

Referring to FIG. 4 , an electronic device (eg, the electronic device 101 of FIGS. 1 and 3 ) according to an embodiment includes a personalized audio management module (eg, FIG. 3 ) of a software module implemented by the processor 120 . An operation method for providing the personalized audio information by the personalized audio management module 221 of ) may be performed.

In operation 401, the electronic device according to an embodiment may receive user interaction information in response to execution of a related application capable of acquiring user interaction information. The electronic device may identify the reception of the interaction information as a trigger action (or event) for starting a function for mapping personalized audio. For example, when the event receives at least one of audio, video, image, or text included in the interaction information, executing instructions stored in the memory (eg, the memory 130 of FIG. 1 ) to execute the personalized audio management module It may be an interface execution operation.

In operation 403, the electronic device may identify/determine characteristic information of the target object, which is data included in the received interaction information, by the personalized audio management module executed in response to the event. The target object may mean data such as a photo, GIF, audio, or video, and may be a part of data included in the received interaction information. The specific information may mean at least one data type (eg, mime type) of audio, video, image, and text.

In operation 405, the electronic device may analyze the target object based on the characteristic information identified by the personalized audio management module. The electronic device may analyze the target object for each data type of the feature information, and identify a speaker related to the target object based on the analysis result. In order to identify the speaker, the electronic device uses at least one of a related database (eg, a speaker voice audio information DB, a contact DB, a target object DB, and a description text DB) included in the database module (eg, the database module 350 of FIG. 3 ). 1) may be used to select a speaker identifier corresponding to the analysis result, and a speaker for the target object may be identified by the selected speaker identifier.

In operation 407, the electronic device generates personalized audio information (eg, TTS audio) in response to the identified characteristic information based on the analysis result by the personalized audio management module, and maps the generated personalized audio information to the target object (or mix) to create content to run.

In operation 409, the electronic device may store and execute the personalized audio information generated by the personalized audio management module in a memory. The electronic device may store content (eg, dynamic content) generated by mixing a target object and personalized audio information (TTS audio) in a memory (eg, the memory 130 of FIG. 1 ) by the personalized audio management module. For example, audio information (or file) may be in an audio format supported by the electronic device (eg, wav, mp4, ogg, gsm, dct, flac, au, aiff, vox, raw, aac, atrac, ra, ram, dss, msv, dvf, etc.). Audio information (or file) is TTS audio obtained by converting text into speech through an audio conversion device (not shown), and may be applied to an audio portion of a target object in the form of speech or narration. For another example, when audio information (or file) is transmitted in the form of data to a network through a communication module (eg, the communication module 180 of FIG. 1 ), a format stored in a memory corresponding to a user interface can mean

5 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment; FIG. 6 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment; It is a diagram illustrating an example of a database for an operation method in an electronic device, and FIG. 7A is a diagram illustrating an example of a speaker voice audio signal characteristic spectrum file for an operation method in an electronic device according to an embodiment. 8A is a diagram illustrating an example of an operation method in an electronic device according to an embodiment, and FIG. 8B is a diagram illustrating an example of a database for an operation method in an electronic device according to an embodiment.

Referring to FIG. 5 , in operation 501 , the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment receives a related application (eg, an interview, a phone call, or a music application) by executing it. Audio information (eg, file, signal, data, or content) may be acquired as a target object included in the interaction information.

In operation 503, the electronic device identifies an audio type (eg, human voice, music, natural sound, and/or machine sound) included in the acquired audio information, and an audio corresponding to at least one user's voice in the acquired audio information Data can be classified. For example, as shown in FIG. 6 , the electronic device may select only a speech using a sound recognition deep learning model 610 .

In operation 505, the electronic device may recognize at least one speaker by analyzing only the voice classified from the separated audio data. For example, as shown in FIG. 6 , the electronic device may identify a speaker identifier (ID) using a speaker recognition deep learning model 620 . For example, an electronic device classifies speech through a synthetic neural network (CNN) model trained with sound train data composed of sounds other than speech (human voice), including sounds from the natural environment, and Only the classified sounds are selected again through a synthetic neural network (CNN) model trained with the voice data of several people, and the speaker ID is distinguished by selecting the voice of the most similar speaker. There may be various types of audio signal features, for example, a mel spectrum obtained through a mel frequency cepstral coefficient (MFCC) algorithm may be used. As another example, prosody information including the speaker's base frequency, loudness, and base pitch may be stored.

In operation 507, the electronic device may store or update information on the identified speaker in a designated DB (eg, the speaker's voice audio information DB). For example, the electronic device obtains the parameters obtained through extraction of speech features for each speaker in the speaker voice audio information DB 710 as shown in FIG. 7A and the speaker voice audio signal characteristic spectrum file 721 as shown in FIG. 7B . and 723). The speaker voice audio information DB 710 as shown in FIG. 7A maps information related to individualized voice characteristics for each speaker and may be stored in the form of a table, and a speaker identifier (ID), audio signal characteristic value, and audio for each speaker. A signal characteristic value graph path (Path), a contact identifier (ID), and a tone identifier (ID) may be mapped. The speaker's voice audio information DB 710 may be linked with the speaker's voice audio signal characteristic spectrum files 721 and 723 as shown in FIG. 7B . The speaker voice audio information DB 710 is downloaded from the market, such as the personal table 711, the voice information of another user who is the user's acquaintance extracted from the user's voice recording audio, and the market table 713. It is possible to store voice information of one other speaker. Among the voice information of the personal table 711 , an ID of a contact may be used when selecting a speaker by designating a contact in the electronic device, and a tone ID may be used when selecting an implicit speaker.

In operation 509, the electronic device may perform speech to text (STT) for each section. As shown in FIG. 8A , when the acquired audio information (or audio file) 801 is audio information that needs to be summarized, the electronic device divides the audio information for each section, and sub-audio information (or sub-audio information) for each divided section, as shown in FIG. 8A . Audio files) 803 may convert speech to text (STT) 805 .

In operations 511 and 513 , the electronic device may generate an STT summary and descriptive text. For example, the electronic device may generate summary text (eg, description text) by summarizing the text files 807 converted for each section using a text summary deep learning model 809 . The electronic device may map the generated summary text with a descriptive text identifier (ID) and a target object identifier (ID) and store it in the descriptive text DB 811 . The descriptive text DB 811 may be linked with the target object DB as shown in FIG. 8B through the target object identifier (ID). The target object DB may include a table in which a target object identifier (ID), a target object data type (mime type), a target object file system path, a contact identifier (ID), and a description text ID are mapped. Here, since the operation of speech to text (STT) 805 may use a known technology, a detailed description thereof will be omitted. A speaker identifier (ID) is a personalized voice (eg, at least one of a girlfriend's voice, a family voice, a child's voice, your own voice, a celebrity's voice, or a voice extracted from audio) to be used when mapping personalized audio information to a target object. It may be an identifier indicating a mapped speaker.

9 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.

In operation 403 and 405 of FIG. 4 , the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment determines that the video analysis module ( TOAVM) (eg, the video analysis module 322 of FIG. 3 ) may analyze video information (or file) of the target object.

Referring to FIG. 9 , in operation 901 , in operation 901 , the electronic device uses video information ( e.g. files, signals, data or content).

In operation 903 , the electronic device may analyze a scene from the obtained video information. For example, the electronic device may use a video analysis module (TOAVM) to convert video information (or files) through a long-term short-term memory model (LSTM) (not shown) using a video summary deep learning model. A scene (key video frame) can be output in binary format. For example, a video summary model may first decompress a video file to generate a set of video frames, and input the video frames in order to the LSTM to output 1 if it is a main scene, and 0 otherwise. . Long and short-term memory model (LSTM) is supervised learning, and after learning main and non-main scenes for a number of video files, it has parameters for the learning results, can be inferred. The electronic device may identify main scenes (or key frames) based on the scene analysis result. The electronic device maps the key frames to the decompress result based on the binary output result (eg, 1) of the key frames to generate a key video frame image. can be extracted.

In operation 905 , the electronic device may generate descriptive text corresponding to key frames extracted by analyzing the video information. The electronic device may map and store the video information of the target object and the generated description text. For example, the electronic device may store a table in which video information of the target object and the generated description text are mapped in the target object DB as shown in FIG. 8B .

10 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.

In the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment, in operations 404 and 405 of FIG. 4 , if the characteristic information of the target object is image information (or content), the image analysis module ( TOAIM) (eg, the image analysis module 323 of FIG. 3 ) may analyze image information of the target object.

Referring to FIG. 10 , in operation 1001 , the electronic device uses an image as a target object included in interaction information received by execution of a related application (eg, an application related to photographing, album execution, image editing, or image transmission/reception). Information (eg, files, signals, data, or content) may be obtained.

In operation 1003 , the electronic device may analyze scenes of the acquired image information by the image analysis module TOAVM (eg, the image analysis module 323 of FIG. 3 ). For example, the electronic device may use an image caption deep learning model including a convolutional neural network and a long-term memory neural network (LSTM) for scene analysis. For example, a convolutional neural network divides an image into small windows, obtains the sum of pixel values for each window, obtains window characteristic values, and sets a constant value to a set of window characteristic values. Filter characteristic values may be obtained by applying a size filter. The convolutional neural network rearranges filter feature values in one dimension to generate a fully connected layer value, and through supervised learning, It is possible to store the parameter values obtained by training using the label as the training data (image train data).

In operation 1005 , the electronic device may generate descriptive text for each scene of image information by the image analysis module TOAVM (eg, the image analysis module 323 of FIG. 3 ). The electronic device may infer what type of image the user's image is by using an image caption deep learning model, and obtain labels (or words) through image inference. have. The electronic device may arrange words using a long-term memory neural network (LSTM) for correct sentences, receive each generated word as an input, and rearrange the words in an order close to a natural language. Long and short-term memory neural network (LSTM) is a model with parameters for natural language sequence through supervised learning. A set of words received as input can be inferred into a correct sentence structure. The image caption model can generate descriptive text for image information through image caption deep learning modeling.

In operation 1007 , the electronic device may map and store image information of the target object and the generated description text. For example, the electronic device may store a table in which image information of the target object and the generated description text are mapped in the target object DB as shown in FIG. 8B .

According to an embodiment, in operations 403 and 405 of FIGS. 4 , in the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ), if the characteristic information of the target object is text information, a text analysis module (TOATM) ( Example: The text information of the target object may be analyzed by the text analysis module 324 of FIG. 3 ), and typos or profanity may be corrected in the analysis result. The electronic device may map the corrected text information with the personalized audio information by a personalized content mapping module (eg, the personalized content mapping module (PCMM) 330 of FIG. 3 ).

11 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.

Referring to FIG. 11 , in operation 1101 , the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment performs at least one speaker (or speaker identifier) as in operation 407 of FIG. 4 . can be selected. In operation 1103 , personalized audio information that is audio obtained by converting text into speech (TTS) into voice corresponding to at least one speaker may be generated based on the audio signal characteristic value mapped to the selected at least one speaker.

In operation 1105 , the electronic device may identify and reprocess the TTS audio track mapping information. When the target object is a moving picture including time information, the electronic device may re-process TTS audio tracks by mapping a plurality of TTS audios to a key video frame. The plurality of TTS audio files may be reprocessed into a TTS audio file whose duration is adjusted, and may be applied to an operation of mixing a target object and personalized audio information that is TTS audio.

In operation 1107, the electronic device may generate content to be provided by mixing personalized audio information that is the generated TTS audio and a target object (or original content).

12A to 12E are diagrams illustrating an example of speaker selection in an operating method in an electronic device according to an exemplary embodiment.

According to an embodiment, when the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) maps (or mixes) the personalized audio information generated to the target object, at least one speaker (or speaker identifier) may be selected (eg, operation 407 of FIG. 4 ). The electronic device maps through at least one of an explicit speaker selection method, a quoted phrase speaker selection method, or a speaker selection method by tone using DBs designated by a personalized content mapping module (PCMM) (eg, the personalized content mapping module 330). It is possible to select at least one speaker (or speaker identifier) to be used. For example, if the sender of the target object is explicit, the electronic device may use the explicit speaker selection method. If the sender of the target object is not explicit, the electronic device may use one of a quoted phrase speaker selection method or a speaker selection method for each tone as an implicit speaker selection method.

Referring to FIG. 12A , the electronic device according to an embodiment searches for a contact identifier (eg, C1 ) corresponding to a sender in the contact DB 1220 using an explicit speaker selection method, and in the target object DB 1210 , A target object identifier (eg, O1) corresponding to the retrieved contact identifier (eg, C1) may be identified. The electronic device may identify the contact identifier (eg, C1) and the mapped speaker identifier (eg, S1) in the speaker voice audio information DB 1230 . The electronic device may select the identified speaker identifier (eg, S1 ) as a speaker for mapping with the target object. The electronic device generates mapping information (or table) 1241 in which the searched speaker identifier (eg, S1) and the target object identifier (eg, O1) are mapped, and stores the generated mapping information 1241 in the speaker mapping table DB 1240 ) can be stored in For example, the generated mapping information 1241 may include a mapping table identifier (ID) (eg, M1), a target object identifier (ID) (eg, O1), a start offset (eg 0), and an end offset (eg, end ) and a speaker identifier (ID) (eg, S1). For example, the target object DB 1210 may include a target object ID, a data type of the target object (eg, mime type), a target object file system path, a contact ID, and a description text ID as mapping information of the target object. have. For example, the contact DB 1220 may include contact ID, name, contact information, and email as contact mapping information. For example, the speaker voice audio information DB 1230 is mapping information related to the generation of personalized audio information, and may include a speaker ID, an audio signal characteristic value, an audio signal characteristic value graph path, a contact ID, and a tone ID. can

12B and 12C , if the sender is not explicitly specified, the electronic device according to an embodiment uses a quoted phrase speaker selection method to provide a target object (eg, a text message or a voice message) of the received interaction information. The descriptive text (eg mike asks the due data) corresponding to the included name to search for (eg mike) is retrieved from the descriptive text DB 1250, and the retrieved descriptive text (eg mike asks the due data) is retrieved. A mapped target object identifier (eg, O1) may be identified from mapping information (or table) 1251 including For example, the descriptive text DB 1250 may include descriptive text ID, target object ID, and descriptive text as mapping information of descriptive text. The electronic device identifies the contact identifier (eg, C1) mapped to the target object identifier (eg, O1) identified in the target object DB 1210, and the contact identifier (eg, C1) identified in the speaker voice audio information DB 1230 A speaker identifier (eg, S1) mapped to C1) may be identified. The sender (eg, sally) may be identified in the contact DB 1220 based on the contact identifier (eg, S1) mapped to the target object identifier (eg, O1). The electronic device may identify a contact ID (eg, C2) mapped to a name (eg, mike) to be searched by the electronic device in the contact DB 1220 . The electronic device may identify and select a caller (eg, sally) and a speaker (eg, mike) of the searched name based on the identified contact IDs (C1 and C2). The electronic device generates mapping information (or tables) 1241 and 1242 in which a target object identifier (eg, O1) is mapped to each of the identified speaker identifiers (eg, S1 and S2), and the generated mapping information (or tables) 1241 and 1242 may be stored in the speaker mapping table DB 1240 .

12D and 12E , when the sender is not explicit or does not exist in the contact DB, the electronic device according to an embodiment uses a speaker selection method for each tone, and a target object of the received interaction information (eg: Since there is no information matching the text message or voice message), a sentimental analysis model 1260 may be used for tone analysis for tone analysis. The electronic device acquires an analysis result (eg, serious, serious, and official) 1261 through the emotion analysis model 1260 for tone analysis, identifies a speaker based on the obtained analysis result 1261, and identifies the identified Mapping information (or table) corresponding to the speaker may be generated. For example, the electronic device identifies a target object ID (eg, O1) mapped to a data type (eg, text) of characteristic information of the target object, and descriptive text mapped to the identified target object ID (eg, O1) ID (eg D1) can be identified. The electronic device may identify the descriptive text ID (eg, D1) and the descriptive text (eg, [safety character] *Month* hour COVID19 status) mapped to the target object ID (eg O1) in the descriptive text DB 1250. have. The electronic device searches the tone information DB 1270 for the analysis result 1261 obtained through tone analysis, and a tone identifier (eg, serious, serious) mapped to the tone (eg, serious, serious) retrieved from the speaker's voice information DB 1230 . M1) can be identified. The electronic device may identify a speaker identifier (eg, S1) mapped to a tone identifier (eg, M1) identified in the speaker's voice information DB 1230 . The electronic device generates mapping information (or table) 1241 in which the searched speaker identifier (eg, S1) and the target object identifier (eg, O1) are mapped, and stores the generated mapping information 1241 in the speaker mapping table DB 1240 ) can be stored in For example, the emotion analysis model 1260 may calculate a weight by generating a corpus from each word included in the text file, vectorizing the words (word2vec), and passing them through multiple layers of neurons. The electronic device may acquire a tone by learning a received text message or a text file of an audio, image, or video through an emotion analysis model.

13 is a diagram illustrating an example of an operation method in an electronic device according to an embodiment, FIG. 14A is a diagram illustrating an example of an operation method in an electronic device according to an embodiment, and FIG. 14B is an embodiment It is a diagram illustrating an example of a database for an operating method in an electronic device.

According to an embodiment, when the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment generates personalized audio information that is TTS audio, as in operation 407 of FIG. 4 , , audio track mapping information identification and reprocessing operations may be performed. The electronic device based on the mapping information (or table) 1241 corresponding to the identified speaker, an audio signal characteristic value (eg, M1) mapped to the speaker identifier (eg, S1) in the speaker's voice audio information DB 1230 Based on the identified audio signal characteristic value (e.g., M1), the descriptive text of the video object, the descriptive text of the image object, or the TTS audio of the text of the text object is generated by TTS audio. can do. When the target object is a moving picture including time information, the electronic device may re-process TTS audio tracks by mapping a plurality of TTS audios to a key video frame. The plurality of TTS audio files may be reprocessed into a TTS audio file whose duration is adjusted, and may be applied to an operation of mixing a target object and personalized audio information that is TTS audio. When the target object is a moving picture, the electronic device may map a plurality of TTS audios to a key video frame. When the electronic device maps TTS audio to a plurality of main scenes, as shown in FIG. 13 , conditions (eg, case1, case2) between the TTS audio and the duration of a video frame and case 3), the TTS audio tracks may be reprocessed. For example, the first condition (case 1) may be a condition in which a duration of a scene of a video frame and a duration of a TTS audio match. For example, the second condition (case 2) may be a condition in which the duration of the scene of the video frame and the duration of the TTS audio do not match. For example, the third condition (case 3) may be a condition in which the duration of a scene of a video frame matches the duration of TTS audio, but different TTS audios overlap each other.

Referring to FIG. 14A , the electronic device may perform a mixing operation by reprocessing a plurality of TTS audio files into a TTS audio file whose duration is adjusted.

In operation 1401, the electronic device may initialize the video object and the pointer of the first table (description text relation table) 1431 as shown in FIG. 14B to 0.

In operation 1403, the electronic device may inquire an item (or mapping information) pointed to by the pointer in the video object and descriptive text relationship table. For example, when the pointer is 1 as shown in FIG. 14B , the electronic device may identify a start time and an end time mapped to a text ID (eg, 1) in the first table 1421 .

In operation 1405, the electronic device may calculate a duration for which a description of an item is satisfied (eg, desc = item.desc_end_time = item.desc_start_time).

In operation 1407, the electronic device calculates the duration of the TTS audio corresponding to the item by using the tables 1431 and 1432 and the file system 1433 as shown in FIG. 14B (eg: TTS_audio_duration = audio_paly_time(item.textid.path)).

In operation 1409 and operation 1411 , the electronic device may insert a silence equal to a value obtained by subtracting the audio duration from the depiction duration and fade out processing as it identifies that the depiction duration is greater than the TTS audio duration. have.

In operations 1413 and 1415 , the electronic device may set a value obtained by subtracting the depiction duration from the TTS audio duration as a threshold value as the electronic device identifies that the TTS audio duration is greater than the depiction duration at the same time as operation 1409 . In operation 1417, the electronic device may divide the TTS audio into small sections and calculate audio energy for each section. In operation 1419, the electronic device may align the audio energy for each section and remove a section having the smallest calculated value, and then accumulate audio energy values for each section. In operation 1421 , if the accumulation amount is less than the threshold value, the electronic device continues to operation 1417 ; otherwise, in operation 1423 , the pointer is a last item among items included in the video object and descriptive text relationship table 1431 . recognition can be confirmed. As a result of the check, if it is smaller than the last item (or item value), in operation 1425 , the electronic device increases the pointer by 1 and then performs operation 1403 . As a result of checking in operation 1419, if the pointer is not smaller than the last item (or item value), the operation may be terminated.

15 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.

Referring to FIG. 15 , the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment targets personalized audio information that is generated TTS audio as in operation 407 of FIG. 4 . When generating content by mapping with an object, a content creation operation may be performed differently for each data type of feature information (eg, text, image, or video).

When the electronic device identifies the target object as a text object in operation 1501 and in operation 1503 , the electronic device may generate dynamic content by mixing the generated TTS audio and the target object.

When the electronic device identifies the target object as the image target object in operation 1505, the electronic device arranges the image according to the TTS audio in operation 1507 to 1513, and then executes each frame of the moving picture in the video module (not shown) of the video encoder. By mapping the images arranged in , TTS audio may be mixed with the audio part of the moving picture, and the arranged images and TTS audio may be encoded. In operation 1521 , the electronic device may generate dynamic content in the form of a moving picture. Here, the electronic device may arrange the images based on the audio description for each region of the original image.

When the electronic device identifies the target object as a moving picture object in operation 1515, the electronic device mixes the audio part of the moving image object and TTS audio in operations 1517 to 1519, and encodes the mixed content in operation 1521 to create a dynamic video format You can create content.

Referring to FIG. 16 , when arranging images as in operation 1507 of FIG. 15 , in operation 1601 , the electronic device initializes the pointer of the image object and descriptive text relationship table 1711 as shown in FIG. 17 to 0. can

In operation 1603 , the electronic device may inquire the item pointed to by the pointer (or mapping information included in the table 1711 ) in the image object and description text relation table 1711 as shown in FIG. 17 .

In operation 1605, the electronic device calculates the duration of the TTS audio corresponding to the item based on the descriptive text information table 1712 and the file system 1713 as shown in FIG. 17 (eg, TTS_audio_duration = audio_play_time(item. textid.path)).

In operation 1607, the electronic device may copy the image to the video frame area for the duration of the TTS audio.

In operation 1609 , the electronic device may determine whether the pointer is smaller than a last item (or item value) among items included in the image object and descriptive text relation table 1711 . As a result of the check, if the pointer is smaller than the last item, in operation 1611 , the electronic device may increase the pointer by 1 and then perform operation 1603 .

In operation 1609, if it is checked that the pointer is not smaller than the last item, the operation may be terminated.

According to an embodiment, as shown in FIG. 18 , the electronic device may arrange the images based on the audio description for each region of the original image. For example, the electronic device arranges the images corresponding to the first image coordinate area ((start_x1, start_y1) to (end_x1, end_y1) area) based on the original image, and describes the images arranged in the first image coordinate area. The first TTS audio (audio description1) may be obtained as a result of arranging the first description text (text description1) and converting the first description text (text description1) into audio through TTS. The electronic device may generate a moving picture by copying the image of the first image coordinate area according to the first TTS audio (audio description1). For example, the electronic device arranges images corresponding to the second image coordinate area (area (start_x2, start_y2) to (end_x2, end_y2)) based on the original image, and describes the images arranged in the second image coordinate area. A second TTS audio (audio description1) may be obtained as a result of arranging the second description text (text description1) and converting the second description text (text description2) into audio through TTS. The electronic device may generate a moving picture by copying the image of the second image coordinate area in accordance with the second TTS audio (audio description1).

19 is a diagram illustrating an example of an operating method in an electronic device according to an embodiment.

Referring to FIG. 19 , when the electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) according to an embodiment generates content in operation 407 of FIG. 4 , in the case of a video target object, It may have an audio part, and may perform an operation of mixing TTS audio and an audio part of a moving picture (video object).

In operation 1901 , the electronic device may initialize the pointer of the video object and descriptive text relationship table 1431 to 0 as shown in FIG. 14B .

In operation 1903 , the electronic device may inquire an item (or mapping information) pointed to by the pointer from the video object and description text relationship table 1431 .

In operation 1905, the electronic device may inquire a target item to be mapped of the video target object (eg, target item = desc_start_time to desc end time of item).

In operation 1907, the electronic device may analyze a voice component of the target item.

In operation 1909, the electronic device may select a voice to be emphasized from among the voice of the target item and the narration voice.

In operation 1911 , the electronic device may increase the volume of the voice to be emphasized and decrease the volume of the remaining audio.

In operation 1913, the electronic device may synthesize the TTS audio of the corresponding section with the audio part of the video object after fade in/out before/after the corresponding section.

In operation 1915 , the electronic device may determine whether the pointer is a last item among items included in the video object and descriptive text relationship table 1431 . As a result of the check, if the pointer is smaller than the last item (or item value), in operation 1917 , the electronic device may increase the pointer by 1 and then perform operation 1903 . In operation 1915, if it is checked that the pointer is not smaller than the last item, the operation may be terminated.

According to the above-described exemplary embodiment, the electronic device receives a text from an acquaintance while driving, for example, and identifies the received text as a target object (eg, text information or audio information) when the TTS is set to read. and the data type of the target object (text or Audio), the speaker can be identified. The electronic device may generate personalized audio information from a voice of an acquaintance based on information mapped to the identified speaker. The electronic device may output personalized audio information corresponding to the text received through the voice of the acquaintance by applying the personalized audio information generated by the voice of the acquaintance to the received text.

As another example, the electronic device generates personalized audio information (TTS audio) to which one's own voice is personalized for text information (handwriting) input through the memo application as the memo application is executed, By mixing personalized audio information (TTS audio) with text information, the text information may be output as personalized audio information of one's own voice.

For another example, the electronic device analyzes features in an audio file that is an interview recording file to emphasize narration among the interview recording files, identifies a separated speaker for emphasizing narration according to the analysis result, and the voice of the separated speaker to create personalized audio information. The electronic device may record and store a summary or emphasis as a narration in the middle of the recording file using the personalized audio information generated by the separated speaker's voice.

As another example, when an image (eg, GIF animation) is generated with a stylus pen (electronic pen), the electronic device may generate image content to which a personalized voice is applied by mapping personalized audio information generated in response to the image to the image. . The electronic device may provide a narration function for mapping a personalized voice to a 10-second image generated from a single-take image.

According to the embodiment as described above, in order to notify the user of the text input by the user by voice, the electronic device selects “user input text” as an alarm sound and inputs the text. When the alarm time comes, the user It is possible to display the inputted text on the screen of the display module, and simultaneously output the text as a generated personalized voice (personalized audio information).

According to an embodiment as described above, as an alarm sound (voice) recording function, the electronic device can directly register a child's or own voice as an alarm sound rather than a hard mechanical sound, and select "Alarm sound > Record" You can record and register the alarm content you want to hear. For example, the electronic device may register the recording of the alarm content as "Mom! It's time for San-e to go to the English language school, get ready soon!" For example, the electronic device may provide an alarm sound sharing and search function, and may set various alarm sounds created by other users as my alarm sounds while sharing interesting alarm sounds that individuals have made.

According to an embodiment, an operation method in an electronic device (eg, the electronic device 101 of FIGS. 1 and 2 ) includes an operation of receiving user interaction information, and characteristic information of a target object based on the received user interaction information. identifying at least one speaker by analyzing the identified characteristic information, generating personalized audio information corresponding to the characteristic information based on the identified at least one speaker, and generating the personalized audio information It may include an operation of generating personalized content by mapping to the target object, and an operation of storing and executing the personalized content.

According to an embodiment, in the method, the target object is an object representing at least one of audio, video, image, and text data included in the user interaction information, and the characteristic information is at least one of audio, video, image, or text. It can contain one data type.

According to an embodiment, the identifying of the at least one speaker may include, when the feature information is identified as an audio type, analyzing audio information of the target object and classifying at least one user voice from the audio information , identifying at least one speaker based on a tone of the classified at least one user voice, wherein the generating of the personalized audio information includes an audio signal characteristic value mapped to the identified at least one speaker Based on , the personalized audio information corresponding to the at least one speaker may be generated.

According to an embodiment, the identifying of the at least one speaker may include, when the feature information is identified as an audio type, analyzing audio information of the target object and dividing the audio information into sections, each section Converting the divided sub-audio information into text, generating descriptive text corresponding to the converted text for each section, and mapping the descriptive texts generated for each section with mapping information stored in the target object database stored in the memory; The comparison may include identifying at least one speaker.

According to an embodiment, in the identifying of the at least one speaker, when the feature information is identified as a video type, the video information of the target object is analyzed and at least one of at least one main scene in the video information is identified. extracting an image of , generating descriptive text corresponding to the scene of the at least one image based on a result of analyzing the scene of the at least one image, and storing the generated descriptive text in the memory as a target object It may include an operation of identifying at least one speaker by comparing the mapping information stored in the database.

According to an embodiment, the identifying of the at least one speaker may include, when the feature information is identified as an image type, analyzing image information of the target object to analyze at least one main scene from the image information , generating descriptive text corresponding to the at least one main scene based on the analysis result, and comparing the generated descriptive text with mapping information stored in a target object database stored in the memory to identify at least one speaker It can include actions.

According to an embodiment, the identifying of the at least one speaker may include, when the characteristic information is identified as a text type, analyzing image information of the target object to identify a typo or profanity in the text information; An operation of generating corrected text information in which the identified typo or profanity is replaced with another language, and an operation of identifying at least one speaker using mapping information stored in the target object database stored in the memory based on the generated descriptive text may include

According to an embodiment, the identifying of the at least one speaker may include, when user information is identified based on an analysis result of the target object, the user using a contact database and a speaker voice audio information database stored in the memory. and identifying the at least one speaker mapped to information.

According to an embodiment, in the identifying of the at least one speaker, when user information is not identified based on the analysis result of the target object, text information included in the target information or at least some data of the target information identifying the descriptive text converted from ; and identifying the at least one speaker mapped to the descriptive text by using a descriptive text database, a target object database, and a speaker voice audio information database stored in the memory.

According to an embodiment, in a non-transitory storage medium storing a program, when the program is executed by a processor, the processor receives user interaction information and a target object based on the received user interaction information. an operation of identifying characteristic information of a , identifying at least one speaker by analyzing the identified characteristic information, an operation of generating personalized audio information corresponding to the characteristic information based on the identified at least one speaker, the personalization and instructions executable to execute an operation of generating personalized content by mapping audio information to the target object, and an operation of storing and executing the personalized content.

In addition, the embodiments disclosed in this document are presented for explanation and understanding of the disclosed and technical content, and do not limit the scope of the technology described in this document. Accordingly, the scope of this document should be construed to include all modifications or various other embodiments based on the technical spirit of this document.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C," each of which may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). may be implemented as software (eg, the program 140) including For example, the processor (eg, the processor 120 ) of the device (eg, the electronic device 101 ) may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided in a computer program product (computer program product). Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. or one or more other operations may be added.

100: network environment 101: electronic device
120: processor 130: memory
160: display module 190: communication module
201: software module 221: personalized audio management module

Claims (20)

In an electronic device,
audio module;
Memory; and
at least one processor electrically coupled to the audio module and the memory;
The at least one processor,
Receive user interaction information,
Identifies characteristic information of a target object based on the received user interaction information,
identifying at least one speaker by analyzing the identified characteristic information,
generating personalized audio information corresponding to the characteristic information based on the identified at least one speaker;
Create personalized content by mapping the personalized audio information to the target object,
An electronic device configured to store and execute the personalized content.
According to claim 1,
The target object represents at least one of audio, video, image, or text data included in the user interaction information,
The electronic device, wherein the characteristic information includes at least one data type of audio, video, image, or text.
The method of claim 1, wherein the at least one processor comprises:
When the characteristic information is identified as an audio type, the audio information of the target object is analyzed to classify at least one user voice from the audio information,
identify at least one speaker based on the tone of the classified at least one user voice,
and generate the personalized audio information corresponding to the at least one speaker based on an audio signal characteristic value mapped to the identified at least one speaker.
The method of claim 1, wherein the at least one processor comprises:
and identify the at least one speaker using a speaker voice audio information database stored in the memory, and identify an audio signal characteristic value mapped to the at least one speaker.
The method of claim 1, wherein the at least one processor comprises:
When the feature information is identified as an audio type, the audio information of the target object is analyzed and the audio information is divided into sections,
converting the divided sub-audio information into text for each section,
generating descriptive text corresponding to the converted text for each section,
at least one speaker is identified by comparing the descriptive texts generated for each section with mapping information stored in a target object database stored in the memory;
and generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the identified at least one speaker.
The method of claim 1, wherein the at least one processor comprises:
When the feature information is identified as a video type, analyzing the video information of the target object to extract at least one image for at least one main scene from the video information,
generating descriptive text corresponding to the scene of the at least one image based on a result of analyzing the scene of the at least one image;
comparing the generated descriptive text with mapping information stored in a target object database stored in the memory to identify at least one speaker;
and generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the identified at least one speaker.
The method of claim 1, wherein the at least one processor comprises:
When the feature information is identified as an image type, analyzing the image information of the target object to analyze at least one main scene in the image information,
generating descriptive text corresponding to the at least one main scene based on the analysis result;
comparing the generated descriptive text with mapping information stored in a target object database stored in the memory to identify at least one speaker;
and generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the identified at least one speaker.
The method of claim 1, wherein the at least one processor comprises:
When the feature information is identified as a text type, the image information of the target object is analyzed to identify a typo or profanity in the text information,
generating corrected text information in which the identified typo or profanity is replaced with another language;
identifying at least one speaker using mapping information stored in a target object database stored in the memory based on the generated descriptive text;
and generate personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the identified at least one speaker.
The method of claim 1, wherein the at least one processor comprises:
The electronic device is configured to identify the at least one speaker mapped to the user information using a contact database and a speaker voice audio information database stored in the memory when the user information is identified based on the analysis result of the target object.
The method of claim 1, wherein the at least one processor comprises:
When user information is not identified based on the analysis result of the target object, text information included in the target information or descriptive text converted from at least some data of the target information is identified,
and identify the at least one speaker mapped to the descriptive text using a descriptive text database, a target object database and a speaker voice audio information database stored in the memory.
A method of operation in an electronic device, comprising:
receiving user interaction information;
identifying characteristic information of a target object based on the received user interaction information;
analyzing the identified characteristic information to identify at least one speaker;
generating personalized audio information corresponding to the characteristic information based on the identified at least one speaker;
mapping the personalized audio information to the target object to generate personalized content; and
and storing and executing the personalized content.
12. The method of claim 11, wherein the method comprises:
The target object represents at least one of audio, video, image, or text data included in the user interaction information,
wherein the characteristic information includes at least one data type of audio, video, image or text.
The method of claim 11, wherein identifying the at least one speaker comprises:
when the characteristic information is identified as an audio type, analyzing audio information of the target object to classify at least one user voice from the audio information;
and identifying at least one speaker based on the tone of the classified at least one user's voice,
The operation of generating the personalized audio information includes:
generating the personalized audio information corresponding to the at least one speaker based on the audio signal characteristic value mapped to the identified at least one speaker.
The method of claim 11, wherein identifying the at least one speaker comprises:
analyzing the audio information of the target object and dividing the audio information into sections when the characteristic information is identified as an audio type;
converting the divided sub-audio information into text for each section;
generating descriptive text corresponding to the converted text for each section; and
and comparing the descriptive texts generated for each section with mapping information stored in a target object database stored in the memory to identify at least one speaker.
The method of claim 11, wherein identifying the at least one speaker comprises:
extracting at least one image for at least one main scene from the video information by analyzing the video information of the target object when the feature information is identified as a video type;
generating descriptive text corresponding to the scene of the at least one image based on a result of analyzing the scene of the at least one image; and
and comparing the generated descriptive text with mapping information stored in a target object database stored in the memory to identify at least one speaker.
The method of claim 11, wherein identifying the at least one speaker comprises:
analyzing at least one main scene from the image information by analyzing image information of the target object when the feature information is identified as an image type;
generating descriptive text corresponding to the at least one main scene based on the analysis result; and
and comparing the generated descriptive text with mapping information stored in a target object database stored in the memory to identify at least one speaker.
The method of claim 11, wherein identifying the at least one speaker comprises:
when the characteristic information is identified as a text type, analyzing image information of the target object to identify a typo or profanity in the text information;
generating corrected text information in which the identified typo or profanity is replaced with another language; and
and identifying at least one speaker using mapping information stored in a target object database stored in the memory based on the generated descriptive text.
The method of claim 17, wherein identifying the at least one speaker comprises:
When user information is identified based on the analysis result of the target object, identifying the at least one speaker mapped to the user information using a contact database and a speaker voice audio information database stored in the memory, Way.
The method of claim 11, wherein identifying the at least one speaker comprises:
identifying text information included in the target information or descriptive text obtained by converting at least some data of the target information when user information is not identified based on the analysis result of the target object; and
identifying the at least one speaker mapped to the descriptive text using a descriptive text database, a target object database and a speaker voice audio information database stored in the memory.
In the non-transitory storage medium storing a program, the program, when executed by a processor, the processor,
receiving user interaction information; identifying characteristic information of a target object based on the received user interaction information;
analyzing the identified characteristic information to identify at least one speaker; generating personalized audio information corresponding to the characteristic information based on the identified at least one speaker;
mapping the personalized audio information to the target object to generate personalized content; and
and instructions executable to execute the action of storing and executing the personalized content.

Images