KR102620762B1 - electronic device providing sound therapy effect using generative AI sound source separation technology and method thereof - Google Patents

Abstract

An electronic device comprising at least one processor that processes memory and information, wherein the processor receives sound source information from a server, inputs the sound source information into a sound source separation artificial intelligence model, and generates at least one processor corresponding to the sound source information. Output frequency source information including one frequency source, apply a binaural beat to at least one frequency source among the identified at least one frequency source, and apply a binaural beat to at least one frequency source to which the binaural beat is applied. Sound information generated based on this can be output.

Description

Electronic device providing sound therapy effect using generative AI sound source separation technology and method thereof}

The present disclosure relates to an electronic device and method for providing a sound therapy effect using generative AI sound source separation technology. More specifically, using an artificial intelligence model, sound sources are separated according to the type of source, and the separated sound source is separated. It relates to an electronic device and method that can apply binaural beats and three-dimensional sound to a source.

Sound source separation is a technology that separates multi-source music consisting of multiple sources (instruments, vocals, sound effects, etc.) from a single sound source signal. In other words, it is about technology to individually extract multiple sounds from one sound source file.

Existing sound source separation methods were usually performed manually by filtering in the frequency domain using Fourier Transform. However, this method cannot guarantee perfect results, as it may not be possible to completely separate the desired sound if one sound source signal consists of multiple sources or is mixed with noise, and it takes a lot of time and money.

In addition, existing sound source processing technologies not only have problems with source quality and accuracy, but also often use a stereo method that simply divides the sound source into left and right sides and thus reduces the three-dimensional effect.

AI sound source separation technology is being developed to overcome the limitations of existing sound source processing technology. This AI sound source separation technology analyzes music data by learning a deep learning model and separates each source based on this. This requires a large amount of data and a learning algorithm, and can increase accuracy compared to existing methods.

Conventional AI sound source separation technology generally uses deep learning algorithms, requires a significant amount of computation to run, uses Tensorflow and Python running on a PC and/or server, requires a GPU, and requires deep learning. It does not work with the frameworks Tensorflow Lite or PyTorch Mobile, so speed may depend on the server and/or PC status. Additionally, because it requires a large amount of memory, there is a problem in that it cannot operate on mobile devices.

One aspect of the disclosed invention seeks to provide an electronic device and a control method for generating AI sound source separation stereoscopic space generation audio that enhances three-dimensional effect by combining AI sound source separation technology and NRTF (Neural Related Transfer Function) technology.

In addition, one aspect of the disclosed invention can provide an arousal effect and a relaxation effect by generating a neural beat by applying the frequency deviation of the separated source.

An electronic device according to an aspect of the disclosed invention includes at least one processor that processes memory and information, wherein the processor receives sound source information from a server, inputs the sound source information into a sound source separation artificial intelligence model, and Output frequency source information including at least one frequency source corresponding to the sound source information, apply a binaural beat to at least one frequency source among the identified at least one frequency source, and the binaural bit is Sound information generated based on at least one applied frequency source may be output.

In addition, determining the source direction corresponding to the source type according to the source type of the output at least one frequency source and applying the binaural bit may be performed based on the determined source direction. Sound information may be generated by applying binaural beats to a frequency source and processing stereophonic sound based on the source direction of at least one frequency source to which the binaural beats are applied.

In addition, the sound source separation artificial intelligence model is trained to output frequency source information including the sound source information and at least one frequency source extracted for each type of source through STFT (Short-Time Fourier Transform). The sound source separation artificial intelligence model may be an artificial intelligence model learned through GEMM (General Matrix Multiplication).

In addition, generating the sound information includes extracting at least one time source in the time domain through STFT (Short-Time Fourier Transform) inverse transformation of at least one frequency source to which the binaural bit is applied, and determining the source direction and Based on the extracted at least one time source, stereophonic sound processing may be performed to generate the sound.

Additionally, the source direction may be a direction with respect to azimuth and elevation angles from a specific reference within a virtual space based on the surround sound.

In addition, the source direction includes a first source direction with an azimuth of 0 degrees to 180 degrees and a second source direction with an azimuth of 180 degrees to 360 degrees, and applying the binaural bit includes the at least one A frequency source may generate a frequency deviation with respect to the first source direction and the second source direction.

In addition, it further includes an input/output unit, wherein the processor, in response to receiving a user input regarding the binaural mode from the input/output unit, converts the at least one frequency source to the received binaural mode. The binaural beat corresponding to can be applied.

Additionally, the at least one frequency source may include at least one of a drum frequency source corresponding to the frequency band of the drum or a bass frequency source corresponding to the frequency band of the bass.

Additionally, the stereophonic sound processing may be performed by multiplying the binaural transfer function corresponding to the source direction and the at least one time source.

Additionally, the processor may update the source direction in response to receiving source direction information about the azimuth angle and elevation angle for the type of source from the input/output unit.

In the method of controlling an electronic device according to an aspect of the disclosed invention, sound source information is received from a server, the sound source information is input into a sound source separation artificial intelligence model, and a frequency including at least one frequency source corresponding to the sound source information is generated. Output source information, identify a source direction corresponding to the source type according to the source type of the output at least one frequency source, and based on processing the output frequency source information, determine the source direction corresponding to the source type. Accordingly, binaural beats may be applied and sound information may be generated by stereophonic sound processing based on the source direction of at least one frequency source to which the binaural beats are applied.

In addition, the sound source separation artificial intelligence model may be trained to output frequency source information including the sound source information and at least one frequency source extracted for each type of source through STFT (Short-Time Fourier Transform). there is.

Additionally, the sound source separation artificial intelligence model may be an artificial intelligence model learned through GEMM (General Matrix Multiplication).

In addition, generating the sound information includes extracting at least one time source in the time domain through STFT (Short-Time Fourier Transform) inverse transformation of at least one frequency source to which the binaural bit is applied, and determining the source direction and Based on the extracted at least one time source, stereophonic sound may be processed and generated.

Additionally, the source direction may be a direction with respect to azimuth and elevation angles from a specific reference within a virtual space based on the surround sound.

In addition, the source direction includes a first source direction with an azimuth of 0 degrees to 180 degrees and a second source direction with an azimuth of 180 degrees to 360 degrees, and applying the binaural bit includes the at least one The frequency source may generate a frequency deviation with respect to the first source direction and the second source direction.

In addition, applying the binaural beat may include, in response to receiving a user input regarding the binaural mode from the input/output unit, switching the at least one frequency source to the binaural mode corresponding to the received binaural mode. Binaural beats can be applied.

Additionally, the at least one frequency source may include at least one of a drum frequency source corresponding to the frequency band of the drum or a bass frequency source corresponding to the frequency band of the bass.

Additionally, the stereophonic sound processing may be performed by multiplying the binaural transfer function corresponding to the source direction and the at least one time source.

One aspect of the disclosed invention is to combine AI sound source separation technology and NRTF (Neural Related Transfer Function) technology to provide an electronic device capable of generating AI sound source separation three-dimensional space generation audio that enhances three-dimensional effect, and a control method thereof. .

1 is a diagram for explaining a system in which a method of operating a device according to an embodiment of the present disclosure can be implemented.
FIG. 2 is a diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.
Figure 3 is a diagram for explaining the source direction in one embodiment.
Figure 4 is a diagram for explaining binaural beats according to an embodiment of the present disclosure.
Figure 5 is an example diagram for explaining a monaural queue according to an embodiment of the present disclosure.
Figure 6 is a diagram for explaining a method of obtaining a binaural transfer function according to an embodiment of the present disclosure.
Figure 7 is a diagram illustrating the structure of the ear.
Figure 8 is a diagram relating to acquisition of a binaural transfer function at the concha location according to an embodiment of the present disclosure.
Figure 9 is a diagram relating to acquisition of a binaural transfer function at the eardrum location according to an embodiment of the present disclosure.
Figure 10 is a diagram comparing the concha binaural transfer function and the eardrum binaural transfer function according to one embodiment.
Figure 11 is a diagram illustrating sound sources at different positions at the same elevation angle according to an embodiment of the present disclosure.
FIG. 12 is a diagram illustrating sound sources at different positions at the same azimuth according to an embodiment of the present disclosure.
13 and 14 are diagrams for explaining an embodiment of an electronic device according to an embodiment of the present disclosure.
Figure 15 is a flowchart for explaining a method of controlling an electronic device according to an embodiment of the present disclosure.

Like reference numerals refer to like elements throughout the specification. This specification does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the disclosed invention pertains is omitted. The term 'unit, module, member, block diagram' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'unit, module, member, block' may be implemented as a single component. , it is also possible for one 'part, module, member, or block' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Throughout the specification, when a member is said to be located “on” another member, this includes not only cases where a member is in contact with another member, but also cases where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the disclosed invention will be described with reference to the attached drawings.

Sound source separation is a technology that separates multi-source music consisting of multiple sources (instruments, vocals, sound effects, etc.) from a single sound source signal. In other words, it is about technology to individually extract multiple sounds from one sound source file.

Existing sound source separation methods were usually performed manually by filtering in the frequency domain using Fourier Transform. However, this method cannot guarantee perfect results as it may not be possible to completely separate the desired sound if one sound source signal consists of multiple sources or is mixed with noise, and it takes a lot of time and money.

In addition, existing sound source processing technologies not only have problems with source quality and accuracy, but also often use a stereo method that simply divides the sound source into left and right sides and thus reduces the three-dimensional effect.

AI sound source separation technology is being developed to overcome the limitations of existing sound source processing technology. This AI sound source separation technology analyzes music data by learning a deep learning model and separates each source based on this. This requires a large amount of data and a learning algorithm, and can increase accuracy compared to existing methods.

Conventional AI sound source separation technology generally uses deep learning algorithms, requires a significant amount of computation to run, uses Tensorflow and Python running on a PC and/or server, requires a GPU, and requires deep learning. It does not work with the frameworks Tensorflow Lite or PyTorch Mobile, so speed may depend on the server and/or PC status. Additionally, because it requires a large amount of memory, there is a problem in that it cannot operate on mobile devices.

The 3D audio generation system 1000 applying artificial intelligence-based binaural beats according to an embodiment of the present disclosure may be an invention to solve the above-mentioned problems. Hereinafter, for convenience of explanation, the 3D audio generation system 1000 that applies artificial intelligence-based binaural beats will be referred to as the main system 1000.

FIG. 1 is a diagram illustrating a system 1000 in which a method of operating a 3D audio generating electronic device applying artificial intelligence-based binaural beats according to an embodiment will be implemented. This system 1000 may be a system for providing services. Unless specifically stated otherwise, services in this disclosure may include services related to sound therapy.

Referring to FIG. 1, the system 1000 according to one embodiment may be implemented in various types of devices. In the description of one embodiment, the system 1000 according to one embodiment may be implemented by the electronic device 100. For example, the system 1000 may be implemented through an electronic device 100 and/or a server 200. In other words, the electronic device 100 and/or the server 200 may perform an operation according to an embodiment based on the system 1000 implemented in each device. Meanwhile, the system 1000 according to one embodiment is not limited to what is shown in FIG. 1, and may be implemented in a wider variety of devices and/or servers. Additionally, the system 1000 may be implemented by one electronic device 100. That is, the method of processing a sound source described in the present disclosure includes processing a sound source stored in an electronic device to generate a new sound source, in addition to a technique of receiving a sound source from a server through communication to be described later and processing the sound source.

The electronic device 100 according to one embodiment may be a user device/smart device owned and/or used by a user and/or customer receiving sound therapy. In this case, an operation according to an embodiment described later may be implemented in the form of an application that can operate on a smart device.

The server 200 according to one embodiment may be a device that performs wireless and/or wired communication with the electronic devices 100 and includes a database with a large storage capacity. For example, the server 200 may be linked with a plurality of electronic devices 100.

Additionally, the server 200 may be, for example, a server that provides sound source information, and may stream (transmit) sound source information requested by the user to the electronic device 100. However, it is not limited to this, and external devices and/or servers that can provide sound source information may be applied.

The system 1000 according to one embodiment may include various modules for operation. The modules included in the system 1000 perform operations specified by the physical device (e.g., the electronic device 100 and/or the server 200) on which the system 1000 is implemented (or included in the physical device). It may be computer code implemented to do this or one or more instructions. In other words, the physical device on which the system 1000 is implemented stores a plurality of modules in the memory in the form of computer code, and when the plurality of modules stored in the memory are executed, the physical device corresponds to the plurality of modules. You can perform specified actions.

FIG. 2 is a diagram illustrating the configuration of an electronic device 100 according to an embodiment.

Referring to FIG. 2 , the electronic device 100 may include an input/output unit 110, a communication unit 120, a database 130, and a processor 140.

The input/output unit 110 may be various interfaces or connection ports that receive user input or output information to the user. The input/output unit 110 can be divided into an input module and an output module, and the input module receives user input from the user. User input can take various forms, including key input, touch input, and voice input. Examples of input modules that can receive such user input include traditional keypads, keyboards, and mice, as well as touch sensors that detect the user's touch, microphones that receive voice signals, cameras that recognize gestures through image recognition, etc. A proximity sensor consisting of an illumination sensor or an infrared sensor that detects the user's approach, a motion sensor that recognizes the user's movements through an acceleration sensor or a gyro sensor, and various other types of input means that detect or receive various types of user input. It is a comprehensive concept that includes all of the following. Here, the touch sensor may be implemented as a piezoelectric or capacitive touch sensor that detects touch through a touch panel or touch film attached to the display panel, or an optical touch sensor that detects touch by an optical method. In addition, the input module may be implemented in the form of an input interface (USB port, PS/2 port, etc.) that connects an external input device that receives user input instead of a device that detects user input by itself. Additionally, the output module can output various information and provide it to the user. An output module is a comprehensive concept that includes a display that outputs images, a speaker that outputs sound, a haptic device that generates vibration, and various other types of output means. In addition, the output module may be implemented in the form of a port-type output interface that connects the individual output means described above.

In more detail, the input/output unit 110 may receive user input regarding the binaural mode from the user. Accordingly, the input/output unit 110 may transmit the received user input regarding the binaural mode to the processor 140.

Meanwhile, here, the binaural mode, as will be described in detail below, is determined, for example, by the difference between the frequency of the source heard in one ear and the frequency of the source heard in the other ear, and is heard by both sides. The mode may be divided (categorized) according to the frequency difference of the main source.

As an example, a display-type output module can display text, still images, and moving images. Displays include liquid crystal display (LCD), light emitting diode (LED) display, organic light emitting diode (OLED) display, flat panel display (FPD), and transparent display. display, curved display, flexible display, 3D display, holographic display, projector, and various other types of devices that can perform image output functions. It is a concept that refers to a video display device in a broad sense that includes everything. This display may be in the form of a touch display integrated with the touch sensor of the input module.

The communication unit 120 can communicate with external devices. Accordingly, the electronic device 100 can transmit and receive information with the external device and/or the server 200 through the communication unit. For example, the electronic device 100 may use the communication unit to communicate with an external device so that information stored and created in the system 1000 is shared.

Here, communication, that is, transmission and reception of data, can be accomplished wired or wirelessly. To this end, the communication department includes a wired communication module that connects to the Internet, etc. through a LAN (Local Area Network), a mobile communication module that transmits and receives data by connecting to a mobile communication network through a mobile communication base station, and a WLAN (Wireless Local Area Network) such as Wi-Fi. A short-distance communication module using an Area Network-type communication method or a WPAN (Wireless Personal Area Network)-type communication method such as Bluetooth or Zigbee, and a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System) ) may be composed of a satellite communication module using a satellite communication module or a combination thereof. Wireless communication technology used for communication may include NB-IoT (Narrowband Internet of Things) for low-power communication. At this time, for example, NB-IoT technology may be an example of LPWAN (Low Power Wide Area Network) technology, and may be implemented in standards such as LTE Cat (category) NB1 and/or LTE Cat NB2, and may be referred to in the above names. It is not limited. Additionally or alternatively, a wireless communication technology implemented in a wireless device according to an embodiment may perform communication based on LTE-M technology. At this time, as an example, LTE-M technology may be an example of LPWAN technology, and may be called various names such as eMTC (enhanced Machine Type Communication). For example, LTE-M technologies include 1) LTE CAT 0, 2) LTE Cat M1, 3) LTE Cat M2, 4) LTE non-BL (non-Bandwidth Limited), 5) LTE-MTC, 6) LTE Machine. It can be implemented in at least one of various standards such as Type Communication, and/or 7) LTE M, and is not limited to the above-mentioned names. Additionally or alternatively, the wireless communication technology implemented in the wireless device according to an embodiment includes at least one of ZigBee, Bluetooth, and Low Power Wide Area Network (LPWAN) considering low-power communication. It may be included, and is not limited to the above-mentioned names. As an example, ZigBee technology can create personal area networks (PAN) related to small/low-power digital communications based on various standards such as IEEE 802.15.4, and can be called by various names.

The database 130 can store various types of information. A database can store data temporarily or semi-permanently. For example, the database includes an operating program (OS: Operating System) for driving the electronic device 100, data for hosting a website, a program for generating Braille, or data about an application (e.g., a web application). etc. can be stored. Additionally, the database may store modules in computer code form as described above.

Examples of the database 130 include hard disk drive (HDD), solid state drive (SSD), flash memory, read-only memory (ROM), random access memory (RAM), etc. This can be. These databases can be provided as built-in or detachable types.

The processor 140 controls the overall operation of the electronic device 100. To this end, the processor 140 may perform calculations and processing of various information and control the operation of components of the electronic device and/or server. For example, the processor 140 may execute a program or application to provide a service. The processor 140 may be implemented as a computer or similar device based on hardware, software, or a combination thereof. In hardware, the processor 140 may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and in software, it may be provided in the form of a program that drives the hardware processor 140. Meanwhile, unless otherwise specified in the following description, the operation of the electronic device and/or server may be interpreted as being performed under the control of the processor 140. That is, when modules implemented in the system 1000 are executed, the modules may be interpreted as the processor 140 controlling the electronic device and/or server to perform the following operations.

In summary, an embodiment may be implemented through a variety of means. For example, an embodiment may be implemented by hardware, firmware, software, or a combination thereof.

In terms of implementation by hardware, the method according to one embodiment includes one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and FPGAs. It can be implemented by (field programmable gate arrays), processor, controller, microcontroller, microprocessor, etc.

In terms of implementation by firmware or software, the method according to one embodiment may be implemented in the form of a module, procedure, or function that performs the functions or operations described below. For example, software code can be stored in memory and run by a processor. The memory may be located inside or outside the processor, and may exchange data with the processor through various known means.

According to one embodiment, a processor-readable storage medium (e.g., a non-volatile processor-readable medium) contains one or more instructions or A program may be stored, and one or more instructions or computer programs, when executed by one or more processors, may cause the one or more processors to perform operations according to an embodiment or implementations.

Below, an embodiment will be described in more detail based on the above technical idea. An embodiment described below may be combined in whole or in part to form another embodiment unless they are mutually exclusive, and this can be clearly understood by those skilled in the art.

The contents of Section 1 described above may be applied to an embodiment described below. For example, operations, functions, terms, etc. that are not defined in an embodiment described below may be performed and explained based on the contents of Section 1.

Unless specifically stated otherwise, in the description of an embodiment, A/B/C may mean A and/or B and/or C.

Unless specifically stated otherwise, in the description of an embodiment, greater than/above A may be replaced with greater than/above A.

Unless specifically stated otherwise, in the description of an embodiment, less than/less than B may be replaced with less than/less than B.

Unless specifically stated otherwise, the electronic device 100 may operate independently and/or based on information transmitted and received based on communication with the server 200. For example, in the description of an embodiment below, the electronic device 100 may provide binaural beats and/or sound therapy services based on binaural beats to a user based on information pre-stored therein. there is. As another example, in the description of an embodiment below, the electronic device 100 provides binaural beats and/or a sound therapy service based on binaural beats to the user based on sound source information received from the server 200. can be provided. As another example, in the description of an embodiment below, the electronic device provides binaural beats and/or a sound therapy service based on binaural beats to the user based on both information pre-stored therein and information received from the server. can be provided.

In the present disclosure, binaural beats may refer to sound effects applied to both ears to generate neural beats. Neural beats are a means for synchronizing brain waves, and brain wave synchronization refers to the phenomenon in which the electrical activity of the brain is synchronized with external stimuli such as auditory or visual signals. In the present disclosure, neural beats generated by applying binaural beats can affect the brain and body by causing changes in states of consciousness, including relaxation, concentration, etc.

The human brain produces electrical activity in the form of brain waves, which can be measured using electroencephalography (EEG) technology. As measured by EEG, specific brain waves are detected in certain states of consciousness, such as delta waves for deep sleep, theta waves for relaxation and meditation, alpha waves for wakeful rest, and beta waves for alertness and concentration. Brain wave frequencies are detected in certain states of consciousness.

When an external stimulus, such as a repetitive sound, is input to the brain at a specific frequency, the brain's electrical activity can become synchronized with the external stimulus. Therefore, brain waves of a specific frequency can be generated in the brain by generating a neural beat using the binaural beat of the present disclosure. In other words, by inputting two different tone sound sources at slightly different frequencies to each ear, the brain can recognize the difference between the two tones and the same beat frequency, thereby synchronizing the brain's electrical activity with the binaural beat frequency.

Below, a specific method for providing a sound therapy service using binaural beats is disclosed. Unless specifically stated otherwise, in the description of an embodiment below, information used/obtained/output/displayed, etc. is used in electronic devices. It is information directly identified/obtained by the electronic device 100 and/or the server 200, information stored in a database included in the electronic device 100 and/or the server 200, or information received from the electronic device 100 and/or other external devices. It may be one or more of the information received by 100 and/or server 200.

Below, an electronic device according to an implementation example of the electronic device 100 according to an embodiment will be described, but other devices that perform similar functions may also be the electronic device 100. For example, electronic devices include terminals, smart phones, laptop computers, tablet PCs, e-book terminals, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), and PCs ( personal computer), etc., but is not limited thereto.

In the description of an embodiment, terms such as three-dimensional sound and three-dimensional sound (sound) may be used interchangeably and, unless specifically stated otherwise, may be understood to have the same meaning. Stereoscopic sound may be related to technology that allows listeners and/or users located in a space other than the space where the sound source occurs to perceive the same sense of direction, distance, and space as the space where the sound source occurs. For example, stereoscopic sound may be generated/provided based on binaural rendering based on a binaural transfer function.

In more detail, the electronic device 100 according to an embodiment of the present disclosure may receive sound source information from the server 200. Here, the sound source information may include, for example, digital music data and/or MP3 files, and music files received through a streaming service.

Meanwhile, the electronic device 100 may include at least one processor 140 that processes various data. By way of example, the processor 140 of the electronic device 100 may include, for example, a learning processor for machine learning, and may train a model composed of an artificial neural network using learning data. The artificial neural network learned here can be referred to as a learning model. A learning model can be used to infer a result value for new input data other than learning data, and the inferred value can be used as the basis for a decision to perform an operation.

The electronic device 100 may input sound source information received from the server 200 into a sound source separation artificial intelligence model and output frequency source information including at least one frequency source corresponding to the sound source information.

More specifically, the processor 140 of the electronic device 100 generates raw data including source information including sound source information and at least one frequency source corresponding to the type of the source (here, the raw data can be understood as learning data). Based on this, you can build a machine learning model by analyzing and processing data. To ensure reliability, the reliability of the constructed machine learning model can be verified by using a predetermined ratio of raw data as inspection data for reliability check. Here, the sound source information may include digital sound source data, etc., as described above, and the type of source is, for example, sound sources such as vocals, drums, bass, and piano. It can refer to the type of each instrument that makes up the. More preferably, the processor of the electronic device 100 divides the sound source information into vocals, drums, bass, and other instruments included in the sound source information, and provides source information including a frequency source corresponding to each extracted instrument. The machine learning model can be built to output. That is, the at least one frequency source is one of the frequency sources corresponding to each instrument, such as a drum frequency source corresponding to the frequency band of the drum, a bass frequency source corresponding to the frequency band of the bass, and a vocal frequency source corresponding to the vocal frequency band. It can contain at least one.

According to an embodiment of the present disclosure, the processor 140 may output (decompose) the sound source information into frequency band sound source information through STFT (Short-Time Fourier Transform) based on the processed sound source information. Accordingly, the processor 140 can extract a frequency source in a frequency band corresponding to the type of source (vocal, drum, bass, etc.). That is, based on processing the sound source information, the processor 140 can extract frequency sources, which are data in a frequency band corresponding to the type of source, as many as the number of sources included in the sound source, depending on the type of source.

Here, extracting the frequency source corresponding to the frequency band may not be processed and extracted in a uniform manner according to the frequency band of the source. That is, drums generally fall into the bass range between 20 Hz and 160 Hz, and kick drums and toms fall into the range between 50 Hz and 100 Hz. Additionally, the bass guitar has a frequency range from 41.2 Hz to 329.63 Hz in E standard tuning for the 4-string, the vocal has a frequency range from 250 Hz to 1 KHz for women, and the frequency range from 80 Hz to 350 Hz for men. Typically has a frequency range between 80 Hz and 630 Hz. Accordingly, it can be understood that since overlapping frequency ranges exist, they cannot be extracted uniformly.

As an example, the processor 140 may convert the sound signal into a complex matrix in the frequency-time domain through STFT (Short-Time Fourier Transform) based on processing the sound source information received from the server 200. . More specifically, the processor 140 can generate a complex matrix by approximating it to the product of a sound source feature vector and a basis vector based on non-negative matrix factorization (NMF). Thereafter, the processor 140 can obtain direction information for each source by applying a normalized spatial covariance matrix to the sound source feature vector generated by approximation. Thereafter, the processor 140 may integrate data corresponding to each source through hierarchical aggregation clustering among sources corresponding to the same direction, and extract a frequency source corresponding to each source type. However, it is not limited to this.

As an example, the processor 140 may utilize a known source extraction method and/or a source extraction method developed in the future.

Meanwhile, the processor 140 may rescue a machine learning model using the frequency source corresponding to each type of source extracted based on the sound source information and the sound source information as input. The machine learning model built accordingly may be named, for example, a sound source separation artificial intelligence model. However, this is an example and is not limited thereto.

Meanwhile, the source information may include, for example, frequency source information and/or time source information. More specifically, the frequency source information may be information that includes at least one frequency source corresponding to the type of at least one source after decomposing sound source information in the time domain into the frequency domain. Additionally, the time source information illustratively includes at least one time source in the time domain through STFT (Short-Time Fourier Transform) inverse transformation of at least one frequency source corresponding to the type of the at least one source. It can be.

The electronic device 100 may perform preprocessing on raw data to perform more efficient learning based on the acquired data. Here, the preprocessing is performed by selecting data for classification learning by the user directly or through an external device on the sound signal in the separated time domain generated through the above-described STFT transformation and inverse transformation of the frequency source extracted for each type of source according to the frequency band of each source. It can mean the process of doing something. However, this is an example and is not limited to this.

Meanwhile, the electronic device 100 may use an encoder and decoder of a neural network model. The electronic device 100 may input training data into at least one encoder and obtain at least one vector value as an encoded result. The encoder and decoder are types of neural network models, each of which is Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Network (CNN). Networks), etc., but is not limited to this.

In another embodiment, the processor of the electronic device 100 may train a learning model through a deep learning algorithm.

Deep learning algorithm is one of the machine learning algorithms and refers to a modeling technique developed from artificial neural networks modeled after human neural networks. Artificial neural networks can be composed of multi-layered hierarchical structures.

An artificial neural network (ANN) includes an input layer, an output layer, and at least one intermediate layer (or hidden layer) (e.g., a kernel) between the input layer and the output layer. It may be structured in a hierarchical structure. Deep learning algorithms can ultimately produce highly reliable results through learning to optimize the weights of the activation function between layers based on this multi-layer structure.

The deep learning algorithm applicable to the processor of the electronic device 100 according to an embodiment of the present application may include, for example, a convolutional neural network (CNN) and/or U-net. More preferably, the U-net algorithm can be applied. However, it is not limited to this. It can be understood that different deep learning algorithms may be applied depending on the embodiment.

Unlike techniques in which a learning process is performed by extracting knowledge from existing data, Convolutional Neural Network (CNN) is characterized by having a structure that extracts features of data and identifies patterns of features. The convolutional neural network (CNN) can be performed through a convolution process and a pooling process. In other words, the convolutional neural network (CNN) may include an algorithm composed of a complex convolutional layer and a pooling layer. Here, a process of extracting features of data (eg, a convolution process (convolution process)) is performed in the convolution layer. The convolution process is a process of identifying features by examining adjacent components of each component in the data and deriving the identified features into one piece. As a compression process, the number of parameters can be effectively reduced. In the pooling layer, a process (for example, a pooling process) is performed to reduce the size of the layer that has undergone the convolution process. The pooling process can reduce the size of data, cancel out noise, and provide consistent features in fine details. For example, the convolutional neural network (CNN) can be used in various fields such as information extraction, sentence classification, and face recognition. Meanwhile, since the convolutional neural network (CNN) is a known technology, detailed description will be omitted below.

Meanwhile, the sound source separation artificial intelligence model learned by the processor 140 may be, for example, a model learned based on GEMM. GEMM (General Matrix Multiplication) is one of the matrix operations and may refer to an operation that multiplies two matrices. In particular, in deep learning, GEMM can be used in fully connected layers and convolutional layers. In these layers, the value must be calculated by connecting each node to all nodes in the previous layer. To do this, you can generate the result by multiplying the input matrix and the weight matrix and then adding the bias vector.

More specifically, a fully connected layer is a layer mainly used in the last step of a deep learning model, and can calculate values by connecting all nodes of the previous layer and all nodes of the current layer. To do this, the input matrix can be multiplied by the weight matrix, and then the bias vector can be added to generate the result. This result represents the probability for each class, and the final classification can be performed by applying the softmax function.

Additionally, the convolutional layer is a layer mainly used in the middle stage of a deep learning model, and can generate a feature map by applying multiple filters to the input image. A filter is a small-sized matrix that can be defined as performing a multiplication operation with some area of the image and summing it, and this process can be GEMM. Meanwhile, here, it can be assumed that the filter is the weight matrix and the image is the input matrix. Accordingly, a bias vector can be added even after applying a filter, and non-linearity can be given by applying an activation function.

Meanwhile, the electronic device 100 may exemplarily include a user terminal. That is, the electronic device 100 includes, for example, Personal Communication System (PCS), Global System for Mobile communication (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), and IMT. (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (WCode Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, SmartPad, tablet PC , laptops, wearable devices, digital signage, etc., may include all types of wired and wireless communication devices capable of input and output, but are not limited thereto.

Accordingly, in general, library frameworks such as TensorFlow LITE and/or PYyTorch MOBILE are utilized to execute artificial intelligence models on user terminals. TENSORFLOW LITE is a lightweight version of TensorFlow and can be used on Android and iOS apps, and PYTORCH MOBILE is a mobile version of PyTorch that runs not only on Android and iOS apps but also on Linux. It can also be used in embedded systems.

The electronic device 100 according to an embodiment of the present disclosure can execute an artificial intelligence model using GEMM.

As an example, in the case of an Android OS environment, the electronic device 100 can use GEMM by applying (installing) the BLAS library. As an example, the BLAS library can be used by porting a library written in C/C++ to Android using the NDK (Native Development Kit) on the Android OS. In particular, among multiple BLAS libraries, OpenBLAS, BLIS, Intel MKL, etc. can be used.

Meanwhile, the electronic device 100 can use the Accelerate framework in the iPhone OS operating system. The Accelerate framework can provide libraries that accelerate linear algebraic computations.

Meanwhile, the electronic device 100 can increase computational processing speed and memory efficiency by using GEMM, and compared to Tensorflow Lite and/or PyTorch Mobile, GEMM is not dependent on the framework and can support any deep learning model. It can provide applicable effects. Additionally, GEMM can be used not only in CPUs and GPUs but also in accelerators such as TPUs and FPGAs, providing high performance and general-purpose performance.

The processor 140 of the electronic device 100 responds to obtaining frequency source information including at least one frequency source according to the type of at least one source input and output to the sound source separation artificial intelligence model, and the output Depending on the source type of at least one frequency source, the source direction corresponding to the source type may be determined.

More specifically, source direction may mean a direction with respect to the azimuth and elevation angles of the time source from a specific reference within a virtual space based on the surround sound. That is, the source direction may be a direction for separating sound source information and providing an effect such as sound signals being heard from different spatial locations depending on the type of source included in the sound source within a virtual space based on stereoscopic sound. However, it is not limited to this, and in another embodiment, the source direction may be used to distinguish one source of the same source type from the left and/or right.

Figure 3 is a diagram for explaining the source direction in one embodiment.

Referring to FIG. 3, in response to obtaining frequency source information including at least one frequency source according to the type of the at least one source, the processor 140 according to an embodiment of the present disclosure determines the type of the source. Accordingly, a predetermined source direction can be identified. However, the source direction is not limited to this, and the source direction can be received from the user through the input/output unit 110.

The predetermined source direction corresponding to the type of source according to an embodiment of the present disclosure may be related to the azimuth angle and elevation angle corresponding to the type of source. More specifically, the predetermined source direction corresponding to the type of source may include FL and FR directions when the source type is vocal, FC and LFE directions when the source type is vocal, and FC and LFE directions when the source type is vocal. In the case of other instruments (other instruments) in the LS and LR directions, it may be in the BLS and BRS directions. However, this is an example and is not limited thereto.

In another embodiment, the predetermined source direction may be a predetermined source direction according to the binaural mode. This may be related to a mapping table that will be described later. In summary, the predetermined source direction is determined by the type of a predetermined source among a plurality of frequency sources included in the sound source information, depending on the binaural mode selected by the user (a mode showing a specific effect among a plurality of binaural modes). It may be the source direction on the mapping table.

In another embodiment, when at least one frequency source is output according to the type of at least one source, the processor 140 may identify a predetermined direction corresponding to the type of the source. However, it is not limited as shown. In other words, it can be understood that the position shown in the drawing is an arbitrarily set direction for convenience of explanation and can be changed to an appropriate position.

In another embodiment, the source directions corresponding to one type of source include the left direction (varying depending on the reference, e.g., right) with an azimuth of 0 degrees to 180 degrees and the right (depending on the reference) direction with an azimuth of 180 degrees to 360 degrees. may include direction (e.g., left). Hereinafter, for convenience of explanation, if the azimuth angle is 0 degrees to 180 degrees, it will be called the left direction, and if the azimuth angle is 180 degrees to 360 degrees, it will be called the right direction.

In another embodiment, the processor 140 receives source direction information about azimuth and elevation angles according to the type of source through the input/output unit 110, and based on the source direction information, determines the type of source. Azimuth and elevation angles can be updated.

More specifically, the above-described source direction information may include, for example, information about azimuth and elevation angles for the type of source to be changed. The source direction information may be a type of source and an elevation angle and azimuth angle corresponding to the type. In this case, the processor 140 can identify two elevation angles and azimuth angles according to the type of received source through left-right symmetry based on azimuth angles of 180 degrees and 360 degrees. Specifically, referring to FIG. 3, in response to receiving the azimuth and elevation angles for the LS direction with respect to the vocal through the input/output unit 110, the processor 140 generates an RS corresponding to the opposite direction of the vocal. Direction can be identified. Accordingly, the processor 140 can identify two source directions based on source direction information including information about one direction corresponding to the type of source. However, it is not limited to this.

In another embodiment, the source direction information received from the user includes information about a first source direction corresponding to an azimuth angle of 0 degrees to 180 degrees and a second source direction corresponding to an azimuth angle of 180 degrees to 360 degrees corresponding to the source type. may include.

Accordingly, the processor 140 identifies two source directions (a first source direction and a second source direction) according to the type of source, and the sound signals output in the two source directions have frequencies of each other. Binaural bits may be applied so that a deviation can be provided. In more detail, the processor 140 of the electronic device 100 may determine at least one signal according to the type of at least one source input and output to the sound source separation artificial intelligence model. Based on processing frequency source information including one frequency source, binaural beats can be applied according to a predetermined source direction corresponding to the type of source.

That is, as described above, the processor 140 can identify the source direction as left or right depending on the type of at least one frequency source. Accordingly, the processor 140 may apply binaural beats by applying the frequency difference between the sound signal in the left direction and the sound signal in the right direction. In addition, as described above, in response to a user input regarding the binaural mode being received through the input/output unit 110, at least one frequency source is used to generate a frequency deviation corresponding to the binaural mode. It may be applying binaural beats.

The processor 140 according to an embodiment of the present disclosure responds to receiving an input regarding one binaural mode among a plurality of binaural modes through the input/output unit 110, It may be related to output of sound and/or sound information including binaural bits corresponding to the mode.

According to one embodiment, each binaural mode may be defined based on a default frequency and a binaural beat frequency (BBF). According to one embodiment, the fundamental frequency and BBF may be defined for generation of binaural beats. According to one embodiment, the basic frequency may be a frequency for a specific ear among the user's two ears. According to one embodiment, BBF is the frequency of the binaural beat, and may be related to the target brain wave frequency range and/or the target brain wave frequency range to be induced by sound therapy corresponding to the binaural beat. .

According to one embodiment, binaural beats are based on different frequencies 1 and 2 determined based on the fundamental frequency and BBF, and may be input to different ears (left and right) of the user, respectively.

As a method of determining different frequencies 1 and 2, there may be an example as follows. However, the following frequency combination is an example, and any method can be implemented to satisfy the BBF for the difference between frequency 1 and frequency 2 input to different ears, and the implementation of the present disclosure is not limited to the examples below. .

<Example Combination 1>

Frequency 1 (Hz) = fundamental frequency

Frequency 2 (Hz) = Fundamental Frequency + BBF, or

Frequency 2 (Hz) = Fundamental Frequency - BBF

<Example Combination 2>

Frequency 1 (Hz) = Fundamental Frequency - a*BBF

Frequency 2 (Hz) = Fundamental Frequency + (1-a)*BBF

At this time, a is a real number between 0 and 1.

Referring back to the example above, if the user is presented with a 300Hz sound in one ear and a 310Hz sound in the other ear, the user's brain will receive and/or recognize it as a 10Hz wave, and the user's brain waves will be adjusted to 10Hz. You can. In this example, 300Hz or 310Hz may be the fundamental frequency and 10Hz may be the BBF. In summary, the processor 140 sets the frequency information according to the type of source included in the sound source information as the basic frequency, and sets the frequency information between the left and right sound signals based on the BBF determined according to the binaural mode selected by the user. Frequency deviation can be created.

In the present disclosure, binaural mode refers to a mode for a user to obtain a specific brain wave effect, and each binaural mode may refer to a mode for outputting a specific effect to the user. Below, each binaural mode and its effects are explained.

According to one embodiment, the correspondence between fundamental frequency, BBF, and source direction according to binaural mode may be predefined. According to one embodiment, based on a specific fundamental frequency and a specific BBF identified based on the type and correspondence of a specific source, binaural beats for a specific binaural mode and sound information including binaural bits are provided. Can be generated/output. According to one embodiment, sound information including these binaural beats may be output.

Meanwhile, the binaural mode according to an embodiment of the present disclosure may be composed of a plurality of binaural modes depending on the degree of frequency deviation to achieve brain wave synchronization as described below. Each binaural mode is an example, and a known frequency deviation and/or a frequency deviation to be known in the future may be applied.

The binaural mode according to one embodiment may include theta wave binaural mode, alpha wave binaural mode, beta wave binaural mode, delta wave binaural mode, and gamma wave binaural mode. The effects of each binaural mode can be defined as follows, but are not limited to this.

Theta (4-8 Hz): Meditation, deep relaxation, creativity

Alpha (8-14 Hz): Focus, productivity

Beta (14-30 Hz): Increased brain activity, increased concentration

Delta (0.5-4 Hz): deep sleep

Gamma (30-50 Hz): Improves cognitive abilities

Accordingly, binaural beats can be provided by applying the frequency deviation corresponding to each binaural mode to the left and right sound signals. In other words, it can be understood that the above-described BBF can correspond to the frequency according to the binaural mode.

The impact of binaural beats on brain waves and/or the resulting therapy effect may be affected not only by BBF but also by fundamental frequency, source direction, etc. Considering this, according to one embodiment, the database 130 may store the correspondence between the basic frequency, BBF, and source direction according to the binaural mode in advance as a mapping table, and the mapping table includes not only the BBF but also the basic frequency , source direction may also be included.

According to one embodiment, in order to obtain a mapping table, big data including experiments and/or experiment results may be acquired, and the mapping table is at least one bar showing the optimal result from these experiments and/or big data. It may represent the correspondence between binaural mode (related to the effect of binaural beats on brain waves), fundamental frequency, BBF, source direction and other parameters.

According to one embodiment, binaural beats and/or sound information including binaural beats may be output as stereoscopic sound and/or non-stereoscopic sound. According to one embodiment, the sound information direction candidates included in the mapping table may be defined as one or more of an azimuth angle or an elevation angle, and the source direction and/or source direction candidates may be expressed as stereoscopic sound. It may be for output binaural beats and/or sound information including binaural beats.

Meanwhile, according to an embodiment of the present disclosure, application of binaural beats may be determined depending on the type of source. Illustratively, in a case where the sound source information includes sources of drums, bass, and vocals, the processor 140 of the electronic device 100 may apply binaural beats to the frequency sources for the drums and/or bass. there is. This may be to maintain the brain wave synchronization effect, although there may be difficulty in the user's recognition of the frequency deviation adjustment for drums and/or bass.

Meanwhile, the binaural beat is a binaural beat applied according to theta wave binaural mode, alpha wave binaural mode, beta wave binaural mode, delta wave binaural mode, and gamma wave binaural mode. It can be.

Figure 4 is a diagram for explaining binaural beats according to an embodiment of the present disclosure.

Referring to FIG. 4, the direction of sound may be related to the direction of the sound source.

Humans can identify the direction of sound based on interaural timing difference and interaural intensity difference. That is, in FIG. 4 shown in top-view, the distance between the sound source and the left ear is greater than the distance between the sound source and the right ear, so the sound waves reaching the left ear and the sound waves reaching the right ear are Differences may occur. In other words, the amplitude of the sound wave reaching the left ear may be smaller than the amplitude of the sound wave reaching the right ear. Additionally, sound waves reaching the left ear arrive slower than sound waves reaching the right ear. From this, humans can identify the direction of sound.

The direction of the sound source may be defined by an azimuth angle measured from a specific reference and/or an elevation angle measured from a specific reference. In Figure 4, only the azimuth angle is shown for convenience of explanation. As described above, since the effect varies depending on the direction of the sound and/or binaural beat, in order to provide optimal sound therapy, the source direction with the best effect of each sound therapy is also defined, and stereoscopic sound is generated based on that. Must be able to.

The processor 140 of the electronic device 100 according to an embodiment of the present disclosure may generate sound information by processing stereoscopic sound based on the source direction of at least one frequency source to which binaural beats are applied.

More specifically, the processor 140 of the electronic device 100 extracts at least one time source in the time domain through inverse STFT (Short-Time Fourier Transform) transformation of at least one frequency source to which binaural beats are applied, , sound information can be generated by processing stereoscopic sound based on the extracted at least one time source and source direction. However, it is not limited to this.

In another embodiment, the processor 140 of the electronic device 100 processes stereoscopic sound based on at least one frequency source to which binaural beats are applied and the source direction, and processes the stereoscopic sound-processed at least one frequency source. Sound information can be generated by extracting at least one time source to which stereoscopic sound is applied through STFT (Short-Time Fourier Transform) inverse transformation.

Stereoscopic audio processing may be performed, for example, based on the source in a time domain state and/or the source in a frequency domain state and source direction. More preferably, it may be appropriate for stereoscopic sound processing to be performed based on the source and source direction in the time domain. This may be because the calculation speed of applying the binaural transfer function in the time domain is faster than in the frequency domain. Accordingly, the electronic device 100 may be able to process stereophonic sound information that is received through streaming more quickly. However, it is not limited to this.

The processor 140 of the electronic device 100 according to an embodiment of the present disclosure may perform three-dimensional sound processing on a sound source. More specifically, the processor 140 may perform three-dimensional sound processing based on the output time source in the time domain and/or the frequency source in the frequency domain and the source direction corresponding to the type of the source. Here, stereoscopic sound processing may be, for example, processing a binaural transfer function corresponding to the source direction by multiplying the time source and/or the frequency source. However, it is not limited to this.

Meanwhile, the binaural transfer function will be described in detail below. The binaural transfer function described below may be used by the processor 140, may be received from the server 200, or may be previously stored in the database 130.

People perceive distance, direction, and space from sound through various clues contained in sound. The clues can be broadly divided into binaural cues and monaural cues.

Binaural cueing refers to recognizing differences in signals coming into both ears, including binaural level difference (Interaural Level Difference, ILD) for level differences in binaural signals, and binaural time difference (ITD) for time differences. , interaural coherence for correlation, etc., and monaural cueing refers to recognizing the characteristics of the signal itself with one ear. Even if the sound is the same, it tends to be reinforced (peak generation) or attenuated (notch generation) in a specific frequency range due to the filtering effect of body characteristics such as the person's head, ear pinna, and shoulders, depending on the altitude at which it is produced. From these changes in frequency characteristics, people can recognize the altitude at which sound occurs.

The binaural transfer function is a function that expresses the correlation between acoustic signals according to physical characteristics such as a person's head, ear pinna, and shoulders, and includes information about binaural and monaural cues. As mentioned above, the monaural cue used to recognize the altitude of a sound appears in a notch-like form in the binaural transfer function. For example, referring to Figure 5, the notch on the binaural transfer function is different for a sound generated from the front (0°) and a sound generated from a higher position (+45°) or lower position (-45°), People use the characteristics of these notches to recognize the altitude of the sound.

By performing binaural rendering using an accurate binaural transfer function, more realistic three-dimensional sound can be provided to the listener.

Therefore, it is common to provide stereoscopic sound by creating one or several standard binaural transfer functions to provide realistic stereoscopic sound to various listeners and performing binaural rendering using it. At this time, convenience of acquisition and Due to ease of use, it is common to place a microphone on a dummy head or a human concha to obtain a binaural transfer function and generate three-dimensional sound based on this.

However, since humans perceive sound through the sound transmitted to the eardrum, a problem arises in that the binaural transfer function obtained at the concha location does not fully capture the characteristics of the sound perceived by humans. On the other hand, acquiring a binaural transfer function at the location of the eardrum requires inserting an instrument close to the person's eardrum, which makes it difficult to acquire the binaural transfer function because the eardrum is torn or the person feels uncomfortable. In particular, in order to generate a standard binaural transfer function suitable for various listeners, binaural transfer functions must be acquired from several people, so obtaining a binaural transfer function at the eardrum location is equivalent to obtaining a binaural transfer function at the concha location. It becomes even more difficult than doing it.

The present disclosure includes generating a standard binaural transfer function using a binaural transfer function obtained from the eardrum location, and generating three-dimensional sound using the generated standard binaural transfer function.

Meanwhile, the binaural transfer function, like the Head Related Transfer Function (HRTF or Head Related Impulse Response, HRIR), is measured in an anechoic room or simulated in an anechoic room situation, and is a transfer function that reflects only body characteristics such as the shape of the head and two ears. It can be a function. Alternatively, the binaural transfer function may be BRIR (Binaural Room Impulse Response) or BRTF (Binaural Room Transfer Function), which is a function that reflects not only the direction of the sound but also the characteristics of the playback space, that is, the head transfer function reflects the characteristics of the playback space. You can. In the present disclosure, binaural transfer function may be used as a term encompassing head transfer function, BRIR, BRTF, etc., and may be used interchangeably with HRTF, BRIR, etc. Additionally, the binaural transfer function may refer to not only the head transfer function, BRIR, and BRTF, but also other transfer functions that perform similar functions to simulate optimal sound by reflecting body and spatial characteristics. By correcting the sound source transmitted through earphones through the binaural transfer function, users can perceive the sound more naturally and realistically.

Figure 6 is a diagram for explaining a method of obtaining a binaural transfer function according to an embodiment.

Referring to FIG. 6, the test subject is located in a predetermined space (room) where a plurality of speakers are installed, and a microphone is located near each of the test subject's ears. Sounds such as impulse, pink noise, and white noise are played through the speaker, and the reproduced sound can be recorded by reaching the microphone directly without being reflected (direct sound) or by being reflected by a wall or some object and reaching the microphone (reflected sound). there is.

When sound is played and recorded for each speaker, the binaural transfer function corresponding to each speaker can be obtained. The position of the speaker can be expressed as a position relative to the subject's position (e.g., azimuth angle, elevation angle, etc.), and thus the binaural transfer function for a specific position relative to the subject. can be acquired. In this specification, a plurality of binaural transfer functions obtained to correspond to a plurality of positions are referred to as a binaural transfer function set. Additionally, since microphones are located in each of the subject's ears, a binaural transfer function corresponding to the left ear and a binaural transfer function corresponding to the right ear are acquired for each location. In this specification, the binaural transfer function corresponding to the left ear and the binaural transfer function corresponding to the right ear are collectively referred to as a binaural transfer function pair. That is, one binaural transfer function set includes n binaural transfer function pairs corresponding to each of n positions, and each binaural transfer function pair includes a binaural transfer function corresponding to the left ear and a binaural transfer function corresponding to the right ear. Contains the corresponding binaural transfer function. For example, in the case of Figure 6, one binaural transfer function set includes 18 binaural transfer function pairs corresponding to each of 18 positions, and each binaural transfer function pair is a binaural transfer function pair corresponding to the left ear. Includes a binaural transfer function and a binaural transfer function corresponding to the right ear.

Figure 7 is a diagram illustrating the structure of the ear.

Referring to FIG. 7, the binaural transfer function can be obtained by recording the sound arriving at or near the concha of the subject's ear (hereinafter referred to as the "concha location"). In this specification, the binaural transfer function obtained by recording the sound reaching the concha location is referred to as the concha binaural transfer function. Accordingly, the Concha binaural transfer function corresponding to the left ear and the Concha binaural transfer function corresponding to the right ear can be collectively referred to as a Concha binaural transfer function pair.

Figure 8 is a diagram relating to acquisition of a binaural transfer function at the concha location according to an embodiment.

Microphones for recording sounds reaching the concha location can be provided in various forms that can be worn by the subject. For example, referring to FIG. 8, the microphone 2 may be provided as an in-ear type that is inserted into the subject's ear hole. In this case, when the subject wears the in-ear type recording device 1 including the microphone 2 by inserting it into the ear hole, the microphone 2 may be located at or near the subject's concha. Accordingly, the sound reproduced from the speaker can be recorded through the microphone 2 at the subject's concha or near the concha.

Additionally, referring to FIG. 8, a binaural transfer function can be obtained by recording a sound reaching the eardrum or near the eardrum of the subject's ear (hereinafter referred to as the "eardrum location"). In this specification, the binaural transfer function obtained by recording the sound reaching the eardrum is referred to as the eardrum binaural transfer function. Accordingly, the eardrum binaural transfer function corresponding to the left ear and the eardrum binaural transfer function corresponding to the right ear can be collectively referred to as an eardrum binaural transfer function pair.

Figure 9 is a diagram relating to acquisition of a binaural transfer function at the eardrum location according to an embodiment.

Microphones for recording sounds reaching the eardrum may be provided in various forms that can be worn by the subject. For example, referring to FIG. 9, the microphone may be provided as a probe microphone type. In this case, the sound that enters the subject's ear can be collected through the tube (4) at the eardrum and recorded through the microphone (3). The tube 4 may be placed in the ear so that the end of the tube 4 in the eardrum direction is spaced apart from the eardrum by a predetermined distance (eg, 1 mm, 2 mm, 3 mm, 4 mm, 5 mm, etc.). The tube 4 may be made of a silicone material such as medical silicone or a plastic material such as medical plastic, but the material of the tube 4 is not limited thereto.

A microphone for recording sound reaching the concha location is easier for the subject to wear than a microphone for recording sound reaching the eardrum. The microphone used to record sound reaching the eardrum must have the end of the tube positioned near the eardrum, which can cause damage to the subject's eardrum. In addition, the microphone must be worn for a long time to acquire the binaural transfer function of the eardrum, making it difficult for the subject to use the microphone. Movement of the tube may cause the tube to move, damaging the eardrum or changing the sound characteristics. Additionally, since the tube must be appropriately deformed according to the shape of the ear hole, separate settings are required for each subject. Therefore, as described above, obtaining a binaural transfer function at the eardrum location becomes more difficult than obtaining a binaural transfer function at the concha location.

On the other hand, while the tympanic binaural transfer function can better express the directionality and externalization (especially anterior externalization) of sound compared to the concha binaural transfer function, sound distortion occurs because it collects sound through a tube, resulting in relatively lower sound quality. This may fall.

Below, we will explain how to generate a standard binaural transfer function using the concha binaural transfer function and the tympanic binaural transfer function. Hereinafter, the standard binaural transfer function is expressed as the hyper binaural transfer function, which is named to emphasize that it is an ultra-precise binaural transfer function that is more precise than the existing binaural transfer function, and is limited by its name. It should not be interpreted or expanded upon. Additionally, the concha binaural transfer function and tympanic binaural transfer function for generating the hyper binaural transfer function may be directly measured as described above, or may be obtained from a known database. Meanwhile, a standard binaural transfer function or a hyper binaural transfer function can be used as a binaural transfer function prepared in advance to provide an optimal transfer function to any user.

According to an embodiment of the present disclosure, a hyper binaural transfer function can be generated by combining the concha binaural transfer function and the tympanic binaural transfer function. At this time, the synthesized concha binaural transfer function and tympanic binaural transfer function may have been acquired from the same subject. Alternatively, the synthesized concha binaural transfer function and tympanic binaural transfer function may be obtained from different test subjects. For example, the concha binaural transfer function may be obtained from a first subject, and the tympanic binaural transfer function may be obtained from a second subject different from the first subject. Alternatively, the synthesized concha binaural transfer function is generated based on a plurality of concha binaural transfer functions obtained from a plurality of subjects, and the eardrum binaural transfer function is a single eardrum binaural transfer function obtained from a specific subject. It may have been created based on .

According to the first embodiment, a hyper binaural transfer function can be generated by replacing a specific frequency region of the concha binaural transfer function with a specific frequency region of the eardrum binaural transfer function. Here, replacing the binaural transfer function may mean replacing the data of the concha binaural transfer function with the data of the eardrum binaural transfer function.

There may be multiple frequency domains that replace the binaural transfer function. For example, the first frequency region of the concha binaural transfer function is replaced by the first frequency region of the eardrum binaural transfer function, and the second frequency region of the concha binaural transfer function is replaced by the first frequency region of the eardrum binaural transfer function. 2 A hyper binaural transfer function can be created by replacing it with the frequency domain.

The frequency region in which the concha binaural transfer function is replaced by the tympanic binaural transfer function may be a frequency region related to the notch.

For example, in the frequency domain, the tympanic binaural transfer function may include a notch and the concha binaural transfer function may not include a notch.

Figure 10 is a diagram comparing the concha binaural transfer function and the eardrum binaural transfer function according to one embodiment.

Referring to Figure 10, in the frequency range of about 7 kHz to 12 kHz, the tympanic binaural transfer function includes three notches (5-1, 5-2, 5-3) and the concha binaural transfer function includes notches. I never do that. Since the notches in the binaural transfer function are related to the sense of direction of sound, in the binaural transfer function as shown in Figure 9, the concha binaural transfer function in the frequency range of 7kHz to 12kHz is replaced with the eardrum binaural transfer function to obtain a frequency of 7kHz. A hyper binaural transfer function including notches 5 can be generated in the frequency range of ~12 kHz, and this can be filtered to the sound source to provide more realistic three-dimensional sound.

For another example, in the frequency domain, the eardrum binaural transfer function may include a greater number of notches than the Concha binaural transfer function. In this case, compared to using only the concha binaural transfer function, a hyper binaural transfer function including a greater number of notches in the frequency domain can be generated by using the eardrum binaural transfer function together.

According to the second embodiment, a hyper binaural transfer function can be generated by mixing a specific frequency area of the concha binaural transfer function and a specific frequency area of the tympanic binaural transfer function. Here, mixing binaural transfer functions may mean mixing data of the concha binaural transfer function and data of the eardrum binaural transfer function at a predetermined ratio. In this specification, the mixing ratio of the concha binaural transfer function and the tympanic binaural transfer function is explained as meaning the ratio of the tympanic binaural transfer function to the concha binaural transfer function, but this is for convenience of explanation. It's just that. The ratio may be 0.5, 1, 1.5, 2, 2.5, 3, 5 or 10, but is not limited thereto.

There may be multiple frequency regions for mixing binaural transfer functions. For example, the concha binaural transfer function and the eardrum binaural transfer function in the first frequency region are mixed at a predetermined ratio, and the concha binaural transfer function and the eardrum binaural transfer function in the second frequency region are mixed at a predetermined ratio. A hyper binaural transfer function can be created by mixing in proportions.

When mixing binaural transfer functions for a plurality of frequency domains, the mixing ratio of the concha binaural transfer function and the tympanic binaural transfer function may be different for each frequency domain. For example, the concha binaural transfer function and the tympanic binaural transfer function are mixed at a first ratio in the first frequency domain, and mixed at a second ratio different from the first ratio in the second frequency domain to create hyper binaural A transfer function can be created. Of course, a hyper binaural transfer function can be created by mixing the concha binaural transfer function and the tympanic binaural transfer function at the same ratio regardless of the frequency domain.

As described in the first embodiment, the frequency region mixing the concha binaural transfer function and the tympanic binaural transfer function may be a frequency region related to the notch, and redundant description thereof will be omitted.

According to the third embodiment, a hyper binaural transfer function can be generated by adding a notch that is the same as or similar to the notch of the tympanic binaural transfer function to the concha binaural transfer function. For example, hyper binaural transmission is achieved by adding the same or similar notch to the concha binaural transfer function by considering the characteristics of the specific notch, such as the frequency value, dB value, and width of the eardrum binaural transfer function. You can create functions. Here, the dB value of the notch may mean the dB value at which the dB value of the notch is lowest. Additionally, the frequency value of the notch may mean the frequency value at which the dB value of the notch is lowest.

The above-described embodiments of generating a hyper binaural transfer function by combining the concha binaural transfer function and the tympanic binaural transfer function are not mutually exclusive, and at least some of the above embodiments can be used in combination with each other. there is. For example, the first frequency domain is synthesized according to the first embodiment, the second frequency domain is synthesized according to the second embodiment, the third frequency domain is synthesized according to the third embodiment, etc. One embodiment may be applied in combination to generate a hyper binaural transfer function.

Meanwhile, a hyper binaural transfer function can be generated by differently combining the concha binaural transfer function and the tympanic binaural transfer function depending on the position of the binaural transfer function.

Figures 11 and 12 are diagrams for synthesizing binaural transfer functions differently for each location according to an embodiment.

Figure 11 is a diagram illustrating sound sources at different positions at the same elevation angle according to an embodiment of the present disclosure.

FIG. 12 is a diagram illustrating sound sources at different positions at the same azimuth according to an embodiment of the present disclosure.

FIG. 11 shows the front position and lateral position in a case where only the azimuth angle is different at the same elevation angle, and FIG. 12 shows the front position and upward position in the case where only the elevation angle is different in the same azimuth angle. 11 and 12, each position is expressed in coordinates of (azimuth angle, elevation angle), and as an example of the front position, the coordinates (0°, 0°) are the user's exact front position, and as an example of the lateral position, the user's exact As an example of the coordinate (+90°, 0°) position, which is the right lateral position, and the upward position, the coordinate (0°, +90°) position, which is the exact upward position of the user, is shown. However, the front position, lateral position, and upward position are shown. It is not limited to this, and a position where the azimuth angle is in the range of -30° to +30° and the elevation angle is in the -30° to +30° range is the front position, the azimuth angle is in the range of +60° to +120°, and the elevation angle is in the -30° range. Positions ranging from ° to +30° are right lateral positions, azimuths ranging from -60° to -120°, positions ranging from -30° to +30° elevations are left lateral positions, azimuths ranging from -30° to +30°. ° range, positions with elevation angles ranging from +60° to +120° would be considered upward positions.

Below, we will describe some examples of generating a hyper binaural transfer function by differently combining the concha binaural transfer function and the tympanic binaural transfer function depending on the position of the binaural transfer function. The examples to be described later are not mutually exclusive, and at least some of the examples to be described later may be used in combination with each other.

As a first example, in the second embodiment, the mixing ratio of the concha binaural transfer function and the tympanic binaural transfer function may vary depending on the position of the binaural transfer function. For example, when generating a hyper binaural transfer function corresponding to a first position, the concha binaural transfer function and the tympanic binaural transfer function in the first frequency region are mixed at a first ratio, and the first In the case of generating a hyper binaural transfer function corresponding to a second location different from the location, the concha binaural transfer function and the tympanic binaural transfer function in the first frequency region are adjusted to a second ratio different from the first ratio. Can be mixed.

At this time, as shown in FIG. 11, the first position may be the user's front position and the second position may be the user's side position. In this case, the first ratio at the front position may be higher than the second ratio at the lateral position. In other words, the hyper binaural transfer function corresponding to the anterior location may have a higher ratio of the tympanic binaural transfer function than the hyper binaural transfer function corresponding to the lateral location. Since the tympanic binaural transfer function can express sound image externalization better than the concha binaural transfer function, it is possible to generate a hyper binaural transfer function that can better express anterior externalization, and this hyper bar By using the binaural transfer function, stereoscopic sound with improved anterior externalization effect can be provided.

Alternatively, as shown in FIG. 12, the first location may be a location in front of the user and the second location may be a location above the user. In this case, the first ratio at the front position may be higher than the second ratio at the upper position. In other words, the hyper binaural transfer function corresponding to the anterior position may have a higher ratio of the tympanic binaural transfer function than the hyper binaural transfer function corresponding to the upper position. Since the tympanic binaural transfer function can express sound image externalization better than the concha binaural transfer function, it is possible to generate a hyper binaural transfer function that can better express anterior externalization, and this hyper bar By using the binaural transfer function, stereoscopic sound with improved anterior externalization effect can be provided.

Alternatively, the first location may be a front location of the user and the second location may be a rear location of the user. In this case, the first ratio at the front position may be higher than the second ratio at the rear position. In other words, the hyper binaural transfer function corresponding to the front location may have a higher ratio of the tympanic binaural transfer function than the hyper binaural transfer function corresponding to the rear location. Since the tympanic binaural transfer function can express sound image externalization better than the concha binaural transfer function, it is possible to generate a hyper binaural transfer function that can better express anterior externalization, and this hyper bar By using the binaural transfer function, stereoscopic sound with improved anterior externalization effect can be provided.

As a second example, in the first embodiment, the frequency region in which the concha binaural transfer function is replaced by the tympanic binaural transfer function may vary depending on the position of the binaural transfer function. For example, when generating a hyper binaural transfer function corresponding to a first position, the concha binaural transfer function in the first frequency domain is replaced with the tympanic binaural transfer function in the first frequency domain, and In the case of generating a hyper binaural transfer function corresponding to a second position different from the first position, the concha binaural transfer function of the second frequency range different from the first frequency range is converted into a tympanic binaural transfer function of the second frequency range. It can be replaced with a transfer function. At this time, as shown in FIG. 11, the first position may be the user's front position and the second position may be the user's side position. Alternatively, as shown in FIG. 12, the first location may be a location in front of the user and the second location may be a location above the user. Alternatively, the first location may be a front location of the user and the second location may be a rear location of the user.

As a third example, in the second embodiment, the frequency region that mixes the concha binaural transfer function and the tympanic binaural transfer function may vary depending on the position of the binaural transfer function. For example, when generating a hyper binaural transfer function corresponding to a first position, the concha binaural transfer function and the eardrum binaural transfer function in the first frequency domain are mixed, and a hyper binaural transfer function different from the first position is used. When generating a hyper binaural transfer function corresponding to position 2, the concha binaural transfer function and the tympanic binaural transfer function of a second frequency range that is different from the first frequency range can be mixed. At this time, as shown in FIG. 11, the first position may be the user's front position and the second position may be the user's side position. Alternatively, as shown in FIG. 12, the first location may be a location in front of the user and the second location may be a location above the user. Alternatively, the first location may be a front location of the user and the second location may be a rear location of the user.

As a fourth example, in the third embodiment, the notch added to the Concha binaural transfer function may vary depending on the position at which the binaural transfer function corresponds. For example, when generating a hyper binaural transfer function corresponding to the first position, a notch having the first frequency value of the tympanic binaural transfer function is added to the concha binaural transfer function, and the first position is added. In the case of generating a hyper binaural transfer function corresponding to a second position different from You can. At this time, as shown in FIG. 11, the first position may be the user's front position and the second position may be the user's side position. Alternatively, as shown in FIG. 12, the first location may be a location in front of the user and the second location may be a location above the user. Alternatively, the first location may be a front location of the user and the second location may be a rear location of the user.

As the hyper binaural transfer function is created using the concha binaural transfer function and the tympanic binaural transfer function, the hyper binaural transfer function is similar to the concha binaural transfer function and the tympanic binaural transfer function. You can have Hereinafter, the similarity between the concha binaural transfer function and the hyper binaural transfer function is referred to as the concha-hyper similarity, and the similarity between the tympanic binaural transfer function and the hyper binaural transfer function is referred to as the tympanic-hyper similarity. Similarity between binaural transfer functions can be calculated using various widely known methods such as mean squared difference similarity, cosine similarity, and Pearson similarity.

Concha-hyper similarity may be different from tympanic-hyper similarity.

The similarity between binaural transfer functions may vary depending on the frequency domain. For example, in the first frequency domain, the concha-hyper similarity may be greater than the tympanic-hyper similarity, and in the second frequency domain, the tympanic-hyper similarity may be greater than the concha-hyper similarity.

According to the first embodiment, when the concha binaural transfer function is replaced with an eardrum binaural transfer function in a specific frequency region, the eardrum-hyper similarity in the specific frequency region may be greater than the concha-hyper similarity.

According to the second embodiment, when the concha binaural transfer function and the tympanic binaural transfer function are mixed at a predetermined ratio in a specific frequency region, the difference between the concha-hyper similarity and the tympanic-hyper similarity is determined according to the mixing ratio. The relationship between large and small may vary. For example, if you mix more tympanic binaural transfer functions than concha binaural transfer functions (e.g., if the ratio exceeds 1, 1.5, 2, 2.5, 3, 5, or 10), the tympanic-hyper similarity is It can be larger than Concha-hyper similarity.

According to the third embodiment, when adding a notch that is the same as or similar to the notch of the tympanic binaural transfer function to the concha binaural transfer function, the tympanic-hyper similarity is greater than the concha-hyper similarity near the frequency value of the notch. You can.

The similarity between binaural transfer functions may vary depending on location. For example, the eardrum-hyper similarity at the first location may be greater than the eardrum-hyper similarity at the second location. For another example, the concha-hyper similarity may be greater than the tympanic-hyper similarity at the first location, and the tympanic-hyper similarity may be greater than the concha-hyper similarity at the second location.

In the first example, if the first ratio at the anterior position is higher than the second ratio at the lateral position, the eardrum-hyper similarity at the anterior position may be greater than the eardrum-hyper similarity at the lateral position.

The hyper binaural transfer function generation method described above may be performed by the electronic device 100 or may be performed from an external device to transmit the binaural transfer function to the electronic device 100.

13 and 14 are diagrams for explaining an embodiment of an electronic device according to an embodiment of the present disclosure.

As described above, the electronic device 100 according to an embodiment of the present disclosure receives sound source information from a server, inputs the sound source information into a sound source separation artificial intelligence model, and selects at least one frequency corresponding to the sound source information. Outputting frequency source information including a source, applying a binaural beat to at least one frequency source among the identified at least one frequency source, and generating the binaural beat based on the at least one frequency source to which the binaural beat was applied. sound information can be output.

More specifically, the electronic device 100 inputs the sound source information into a sound source separation artificial intelligence model, outputs frequency source information including at least one frequency source corresponding to the sound source information, and outputs the at least one output frequency source information. According to the source type of the frequency source, a source direction corresponding to the source type is identified, and based on processing the output frequency source information, a binaural beat is applied according to the identified source direction, and the binaural bit is applied. Sound information may be generated by performing three-dimensional sound processing based on the source direction of at least one frequency source to which a rul beat is applied.

Meanwhile, the processor 140 of the electronic device 100 generates sound information by applying binaural bits for each type of source and finally mixing (synthesizing) the time source for each type of source. .

Referring to Figures 13 and 14, the processor 140 according to an embodiment of the present disclosure can control the size of the volume of the time source according to the type of each source. In more detail, the processor 140 may control the size of the volume of the time source based on the user input received from the input/output unit 110. As shown, the input/output unit 110 is implemented as a touch display, but is not limited thereto.

According to one embodiment, the electronic device 100 may control the volume of all types of sources according to the master volume in response to a user input regarding the master volume being received through the input/output unit 110. there is. Accordingly, the electronic device 100 can output the volume of sound information that is the final mix (synthesis) of time sources for each source type according to the master volume.

According to one embodiment, the electronic device 100 may control the volume corresponding to the type of source in response to a user input regarding the volume according to the type of source being received through the input/output unit 110. there is. In more detail, referring to (a) of FIG. 14, the electronic device 100 responds to a user input regarding the volume of vocals and drums being received through the input/output unit 110, and outputs the time of vocals. Sound information can be output by final mixing (synthesizing) the time source and drum-related time source to correspond to each volume. As shown, an embodiment of controlling the volume for vocals and/or drums is disclosed, but the user receives a signal to arbitrarily control the volume for each source through the input/output unit 110 of the electronic device 100. can receive.

Figure 15 is a flowchart for explaining a method of controlling an electronic device according to an embodiment of the present disclosure.

The electronic device control method shown in FIG. 15 may be performed by the electronic device 100 and/or system 1000 described above. Therefore, even if the content is omitted below, the content described with respect to the electronic device 100 and/or system 1000 may be equally applied to the description of the electronic device control method.

Referring to FIG. 15, the electronic device 100 may receive sound source information from the server 200 (S101).

Additionally, the electronic device 100 may input the sound source information received in step S101 into the sound source separation artificial intelligence model and output frequency source information including at least one frequency source corresponding to the sound source information (S102). .

Additionally, based on processing the frequency source information output in step S102, the electronic device 100 may identify a source direction corresponding to the source type according to the source type of at least one frequency source included in the frequency source information. (S103).

In addition, based on processing the frequency source information output in step S102 and the source direction identified in step S103, the electronic device 100 transmits binaural bits to the at least one frequency source according to the identified source direction. can be applied (S104).

In addition, the electronic device 100 processes stereophonic sound on at least one source to which binaural beats are applied based on at least one frequency source to which binaural beats are applied and a source direction corresponding to the frequency source to generate sound information. can be created (S105).

The user can select the desired mode. For example, when the user selects the first mode, the BBF corresponding to the first mode may be determined and the sound source may be processed. Accordingly, the user can obtain the effect corresponding to the first mode.

The user can select the second mode. For example, when the user selects the second mode, the BBF corresponding to the second mode may be determined and the sound source may be processed. Accordingly, the user can obtain the effect corresponding to the second mode.

The user can select a third mode. For example, when the user selects the third mode, the BBF corresponding to the third mode may be determined and the sound source may be processed. Accordingly, the user can obtain effects corresponding to the third mode.

The first to third modes may include an awakening mode, a relaxation mode, a cognitive enhancement mode, a sleep mode, etc., respectively.

Additionally, users can control the size of each source that makes up the sound source in each mode. In other words, by reducing the volume of the source to which BBF is applied, it is possible to maintain the neural beat effect and achieve minimal auditory interference.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, and optical data storage devices.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which the present invention pertains will understand that the present invention can be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

1000: System
100: electronic device
110: input/output unit 120: communication unit
130: database 140: processor

Claims (20)

In electronic devices,
Memory;
input/output unit; and
Including at least one processor that processes information,
The processor,
Receive sound source information from the server,
Input the sound source information into a sound source separation artificial intelligence model to output frequency source information including at least one frequency source corresponding to the sound source information,
Determine a source direction corresponding to the source type according to the source type of the at least one output frequency source,
Applying a binaural beat to at least one frequency source among the at least one output frequency source,
Output sound information generated by stereophonic sound processing based on the source direction of at least one frequency source to which the binaural beat is applied,
The processor,
In response to receiving source direction information about azimuth and elevation angle for a type of source from the input/output unit, update the source direction,
The at least one frequency source includes at least one of a drum frequency source corresponding to the frequency band of the drum or a bass frequency source corresponding to the frequency band of the bass.
Electronic devices.
delete According to paragraph 1,
The sound source separation artificial intelligence model is,
Learned to output frequency source information including the sound source information and at least one frequency source extracted from the sound source information for each type of source through STFT (Short-Time Fourier Transform),
Electronic devices. According to paragraph 3,
The sound source separation artificial intelligence model is an artificial intelligence model learned through GEMM (General Matrix Multiplication).
electronic device According to paragraph 1,
Generating the sound information includes:
Extracting at least one time source in the time domain through inverse STFT (Short-Time Fourier Transform) transformation of at least one frequency source to which the binaural beat is applied,
Based on the source direction and the extracted at least one time source, producing stereoscopic sound by processing,
Electronic devices. According to clause 5,
The source direction is,
A direction with respect to azimuth and elevation angle from a specific reference within a virtual space based on the surround sound,
Electronic devices. According to clause 6,
The source direction is,
a first source direction having an azimuth of 0 degrees to 180 degrees and a second source direction having an azimuth of 180 degrees to 360 degrees,
Applying the binaural beat is,
Generating a frequency deviation with respect to the at least one frequency source with respect to the first source direction and the second source direction,
Electronic devices. In clause 7,
The processor,
In response to receiving a user input regarding a binaural mode from the input/output unit, applying a binaural beat corresponding to the received binaural mode to the at least one frequency source,
Electronic devices.
delete According to clause 5,
The three-dimensional sound processing is,
A binaural transfer function corresponding to the source direction is multiplied by the at least one time source,
electronic device delete In an electronic device, receive sound source information from a server,
In the electronic device, the sound source information is input into a sound source separation artificial intelligence model to output frequency source information including at least one frequency source corresponding to the sound source information,
In the electronic device, determine a source direction corresponding to the source type according to the source type of the output at least one frequency source,
In the electronic device, applying a binaural beat to at least one frequency source among the at least one output frequency source,
In the electronic device, sound information generated by stereophonic sound processing is output based on the source direction of at least one frequency source to which the binaural beat is applied,
In the electronic device, update the source orientation in response to receiving source orientation information about an azimuth and elevation angle for a type of source from a user,
The at least one frequency source includes at least one of a drum frequency source corresponding to the frequency band of the drum or a bass frequency source corresponding to the frequency band of the bass.
method.
delete According to clause 12,
The sound source separation artificial intelligence model is,
Learned to output frequency source information including the sound source information and at least one frequency source extracted from the sound source information for each type of source through STFT (Short-Time Fourier Transform),
method. According to clause 14,
The sound source separation artificial intelligence model is an artificial intelligence model learned through GEMM (General Matrix Multiplication).
method. According to clause 12,
Generating the sound information includes:
Extracting at least one time source in the time domain through inverse STFT (Short-Time Fourier Transform) transformation of at least one frequency source to which the binaural beat is applied,
Based on the source direction and the extracted at least one time source, producing stereoscopic sound by processing,
method. According to clause 16,
The source direction is,
A direction with respect to azimuth and elevation angle from a specific reference within a virtual space based on the surround sound,
method. According to clause 17,
The source direction is,
a first source direction having an azimuth of 0 degrees to 180 degrees and a second source direction having an azimuth of 180 degrees to 360 degrees,
Applying the binaural beat is,
Generating a frequency deviation with respect to the at least one frequency source with respect to the first source direction and the second source direction,
method. According to clause 18,
Applying the binaural beat is,
In response to receiving a user input regarding a binaural mode from an input/output unit, applying a binaural beat corresponding to the received binaural mode to the at least one frequency source,
method. A computer-readable recording medium on which a program capable of executing the method of claim 12 is recorded.

Images