KR102640557B1 - Apparatus and method for composing music based on brain-computer interface with the theory of music - Google Patents

Abstract

The BCI-based music composition method according to an embodiment of the present disclosure includes at least part of each step being performed by a processor, receiving brain signal data, and inputting the brain signal data into a first classifier based on machine learning to obtain a brain signal. A step of estimating the first sound imagined by the subject from whom the data was acquired, a step of estimating a second sound by using a plurality of past sounds as input to a second classifier based on machine learning, and Based on the first sound and the second sound It may include the step of determining the final note.

Description

Brain-computer interface-based music composition method and device using music theory {APPARATUS AND METHOD FOR COMPOSING MUSIC BASED ON BRAIN-COMPUTER INTERFACE WITH THE THEORY OF MUSIC}

The present disclosure relates to a technology for decoding (deciphering, interpreting) brain signals, and more specifically, to an apparatus and method for composing music by estimating sounds imagined by a subject through decoding of brain signals. It's about.

Brain-computer interface (BCI) is a technology that uses brain signals to control devices or determine the user's intentions and emotions. It mainly determines the user's intentions based on brain signals generated by the brain's electrical activity. It is a technology that recognizes.

Conventionally, it has been mainly studied in the medical field to recognize the intentions of patients who are physically disabled or paralyzed, but is also being studied for application to various fields such as controlling peripheral devices of the general public, such as games. The inherent method of operating a brain-computer interface is to determine intent by analyzing brain waves triggered by the user's imagination in the absence of external stimulation.

In particular, motor imagery, in which a user triggers brain signals by imagining moving his or her body or a controlled device, and interprets (decodes) the evoked brain signals, is being widely studied.

There exists technology for automatically composing music based on deep learning technologies such as conventional artificial neural networks.

Prior art 1 exists in response to a request for composing a specific style from a user, and there exists a technology that configures the harmonic progression of a song using machine learning of an artificial neural network based on codes obtained from a plurality of sound sources. However, in prior art 1, there is no room for intervention in the user's intended song progression.

In another way, there is a technology in Prior Art 2 that receives the user's voice and determines the scale or beat of the scale corresponding to the user's voice to perform composition. However, prior art 2 requires the user to utter a voice and cannot compose music using only the user's imagination.

Therefore, there is a need for a technology that can implement music scores using the user's imagination while complementing the accuracy of brain signal decoding or the imagination of people who lack knowledge of music composition.

Prior Art 1: Republic of Korea Patent No. 10-1886534 (announced on August 9, 2018) Prior Art 2: Republic of Korea Patent Publication No. 10-2012-0047077 (published on May 11, 2012)

An embodiment of the present disclosure provides an apparatus and method for composing music based on BCI by decoding brain signals imagining sounds.

Another embodiment of the present disclosure provides an apparatus and method for supplementing the analysis results of decoding brain signals imagining sounds with music theory based on artificial intelligence.

One embodiment of the present disclosure provides a BCI-based music composition device and method.

The BCI-based music composition method according to an embodiment of the present disclosure includes at least part of each step being performed by a processor, receiving brain signal data, and inputting the brain signal data into a first classifier based on machine learning to obtain a brain signal. A step of estimating the first sound imagined by the subject from whom the data was acquired, a step of estimating a second sound by using a plurality of past sounds as input to a second classifier based on machine learning, and Based on the first sound and the second sound It may include the step of determining the final note.

A BCI-based music composition device according to an embodiment of the present disclosure includes a processor and a memory electrically connected to the processor and storing at least one code executed by the processor, and the memory allows the processor to input input when executed on the processor. The received brain signal data is input to a first classifier based on machine learning to estimate the first sound imagined by the subject for whom the brain signal data was acquired, and a plurality of past sounds are input to the second classifier based on machine learning to estimate the second sound. A code may be stored that causes the note to be estimated and the final note to be determined based on the first note and the second note.

The device and method according to an embodiment of the present disclosure can compose music only by imagining sounds.

The device and method according to an embodiment of the present disclosure can improve decoding accuracy by supplementing the decoding results of brain signals imagining sounds with music theory based on artificial intelligence.

FIG. 1 is a diagram illustrating an environment for performing a BCI-based music composition method or driving a BCI-based music composition device according to an embodiment of the present disclosure.
Figure 2 is an example of decoding results of brain signals imagining different sounds.
Figure 3 is a block diagram showing the configuration of a BCI-based music composition device according to an embodiment of the present disclosure.
Figure 4 is a flowchart for explaining a BCI-based music composition method according to an embodiment of the present disclosure.
5 and 6 are diagrams for explaining a method of determining the final sound according to an embodiment of the present disclosure.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Referring to FIG. 1, an environment for performing a BCI-based music composition device (hereinafter referred to as 'composition device') or method according to an embodiment of the present disclosure will be described.

An environment for performing a device or method for decoding brain signals according to an embodiment of the present disclosure may include a composition device 100 and a brain signal measurement device 200.

The brain signal measurement device 200 is capable of measuring brain signals including electroencephalography (EEG) from a subject, and has a flexible cap divided at regular intervals according to the international 10-20 system. Brain signals can be received from an electrode cap 210 in which a plurality of electrodes are regularly arranged in a position or from a plurality of electrodes arranged on the head of a subject according to the international 10-20 system. The International 10-20 System is an international standard for the placement of electrodes that are attached or glued to the scalp. It uses the distance between bone marks on the head to determine the line system for electrode placement. Electrodes for measuring brain signals can be placed at intervals of 10% or 20% of the total length of these lines.

In this specification, brain signals are explained using an electroencephalogram as an example, but brain signals are electrical and magnetic signals generated from nerve cells in the brain, such as magnetoencephalography (MEG) and electrocorticogram (ECoG), or images taken of the brain. It is an all-inclusive concept.

The brain signal measurement device 200 may be implemented as included in the composition device 100 or may be implemented separately from the composition device 100. If the brain signal measuring device 200 is implemented separately from the composing device 100, the composing device 100 receives brain signals measured from a plurality of electrodes mounted on the subject from the brain signal measuring device 200 through a network or wired/wireless network. It can be received through the interface.

In another embodiment, the brain signal measured by the brain signal measuring device 200 may be a concept that includes the form of an image. For example, it may be an electroencephalogram topography created based on an electroencephalogram. In this specification, brain signals are a concept that includes not only the electrical and magnetic signals themselves generated from nerve cells in the brain, such as electroencephalograms and magnetoencephalograms, but also topography generated based on the electrical and magnetic signals.

In another embodiment, the brain signal may be a brain image captured over a certain period of time, such as fMRI.

The composition device 100 may receive and process the brain signals acquired by the brain signal measurement device 200 to determine (decode) the sound imagined by the subject whose brain signals were measured.

In this specification, sound refers to not only absolute pitch, but also relative pitch, melody including sound flow or sound shape, sound outline, or note flow, etc., and the pitch, beat, and movement of the sound. It includes all changes in sound compared to sound. Therefore, when an object imagines a sound, it may be imagining a change in pitch of the sound compared to the previous sound, and when a classifier based on machine learning estimates the sound, it may estimate the change in pitch of the sound compared to the previous sound or an absolute It involves estimating something related to a sound, estimating a sound or estimating a change in the contour of a sound. If the initial note is determined and sequentially imagined notes are selected continuously, the sequence of these notes will form a certain context, which is, for example, a sequence of notes based on the harmony of Western tonal music. It can be compared to the logic that can estimate a melody. Therefore, the past sound indicates two or more previously selected sounds that can form a musical context, and the current sound imagined by the subject can be determined by the past sounds.

The composition device 100 may decode the sound imagined by the subject by inputting brain signal data obtained by pre-processing the brain signal into a machine learning-based learning model.

Machine learning-based learning models include CNN or R-CNN (Region based CNN), C-RNN (Convolutional Recursive Neural Network), Fast R-CNN, Faster R-CNN, and R-FCN (Region based Fully Convolutional Network). ), a neural network with a YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector) structure, or a SVM (Support Vector Machine) classifier, and the type is not particularly limited. As explained below, some of the classifiers included in the embodiments of the present disclosure process brain signal data and are therefore applicable regardless of the type of learning model.

The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory.

The composition device 100 may receive brain signals measured from the subject or brain signal data pre-processed from the brain signal and input them into a machine learning-based learning model to estimate the sound imagined by the subject.

Referring to Figure 2, this is the result of measuring the FFT amplitude of the brain signal measured in the subject's temporal lobe area for the 3-degree jump ascending/descending, sequential ascending/descending, and homophonic melody progression imagined by the subject. As can be seen in Figure 2, the averages of the group's downward jump (neg2) and upward jump (pos2) were significantly different, confirming that the sound imagined by the subject could be distinguished through brain signals.

The configuration of the composition device 100 according to an embodiment of the present disclosure will be described with reference to FIG. 3 .

The composition device 100 includes a brain signal measurement device 200 or a communication unit 110 for interfacing or communicating with electrodes for measuring brain signals, and the communication unit 110 includes a mobile communication module, a wireless Internet module, and a short-range A communication module may be included as the network interface 111.

Technical standards or communication methods for mobile communication used in mobile communication modules (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution) Transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network built according to Term Evolution-Advanced, etc.).

The wireless Internet module refers to a module for wireless Internet access and may be built into or external to the composition device 100. The wireless Internet module is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), and WiMAX (Worldwide). These include Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced).

The short-range communication module is for short-range communication and includes Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near Field). Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology can be used to support short-distance communication.

The composition device 100 may include an interface unit 120 for providing user input or a notification to the user, and the interface unit 120 may include a button 121, a display 122, and an optical output module such as an LED. Alternatively, it may include an audio output module 123 such as a speaker. The interface unit 120 may include a mechanical input means (or a mechanical key, dome switch, jog wheel, jog switch, etc.) and a touch input means. As an example, the touch input means consists of a virtual key, soft key, or visual key displayed on the touch screen through software processing, or a part other than the touch screen. It can be done with a touch key placed in .

The composition device 100 includes a memory 130, and the memory 130 may store brain signals transmitted from the brain signal measurement device 200 or measured from the subject or brain signal data obtained by preprocessing the brain signals.

In one embodiment, the memory 130 may store optimized model parameters of an artificial neural network included in a trained learning model of the composition device 100.

In another embodiment, the memory 130 may store training data and code for training a machine learning-based learning model.

The composition device 100 may use the processor 180 to apply a trained learning model to a specific input and infer a result value. In another embodiment, the processor 180 may be used to train a machine learning-based learning model. The processor 180 determines optimized model parameters of the artificial neural network by repeatedly training the artificial neural network using various learning techniques, or infers the result value by applying the learning model to the input according to the model parameters of the learned artificial neural network. can do.

Machine learning-based learning models include CNN or R-CNN (Region based CNN), C-RNN (Convolutional Recursive Neural Network), Fast R-CNN, Faster R-CNN, and R-FCN (Region based Fully Convolutional Network). ), a neural network with a YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector) structure, or an SVN-based classifier.

The learning model may be implemented as hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented as software, one or more instructions constituting the learning model may be stored in the memory 130.

The composition device 100 may include a brain signal processing unit 140 that performs pre-processing of brain signals, and the brain signal processing unit 140 may be implemented as part of the processor 180 or may be implemented separately. there is. In one embodiment, the brain signal processing unit 140 may be implemented as a DSP chip or GPU.

In one embodiment, the brain signal processor 140 may perform amplification and filtering and remove noise or artifacts when the brain signal is an electroencephalogram. Alternatively, the brain signal processing unit 140 may perform algorithms such as Fourier analysis and wavelet analysis, and as it is a common configuration for those skilled in the art, detailed description is omitted herein.

When the brain signal is a brain image, the brain signal processing unit 140 may perform pre-processing of the brain image through an image processing algorithm.

The composition device 100 includes a processor 180 that is triggered to determine the sound imagined by the subject based on the subject's brain signals by a code stored in the memory 130. The processor 180 may determine the sound imagined by the subject based on a code or learning model that implements an algorithm for decoding brain signals.

In another embodiment, the composition device 100 includes a processor 180 that is caused to train a learning model that determines the sound imagined by the subject by inputting brain signal data based on the code and training data stored in the memory 130. may include.

With reference to FIG. 4 , a brain-computer interface (BCI)-based music composition method of the composition device 100 according to an embodiment of the present disclosure will be described. The description below assumes that the brain signal measuring device 200 is implemented separately from the composing device 100, but it does not exclude that the brain signal measuring device 200 is implemented together with the composing device 100. Additionally, the explanation below assumes that brain signals are brain waves, but includes all signals that measure or record brain activity or state based on electrical or magnetic signals generated from the brain.

The composition device 100 may receive brain signal data from a brain signal measurement device or receive brain signals from electrodes for measuring brain signals placed on the head of the subject and preprocess them to obtain brain signal data (S110). Alternatively, brain signal data can be obtained by loading the brain signal data from a storage device. The brain signal may be a signal that measures electrical and magnetic signals generated from nerve cells in the brain, such as an electroencephalogram or magnetoencephalogram, or a topography generated based on an electric and magnetic signal, such as an electroencephalogram or magnetoencephalogram. Or it may be Functional Near-infrared Spectroscopy (fNIRS). Alternatively, the brain signal may be a brain image obtained by imaging the brain with imaging diagnostic equipment such as CT, MRI, or fMRI.

If the brain signal is an electroencephalogram, it may be an electrical signal measured from a plurality of electrodes placed on the subject's head according to the international 10-20 system. Brain signals may be signals measured in the Delta, Theta, Alpha, Beta and/or Gamma bands.

Brain signals may be measured in the temporal lobe of the brain.

The following description describes embodiments on the premise that the brain signal is an electroencephalogram, but those skilled in the art will recognize that the present invention can be similarly applied to other cases.

The composition device 100 may extract feature points from at least a portion of brain signal data. Feature points may be ERP (Event Related Potential), spectrum, or FFT amplitude size. ERP may be Visual Evoked Potentials (VEP), Auditory Evoked Potentials (AEP), Somatosensory Evoked Potentials (SEP), Steady State Evoked Potentials (SSEP) or P300.

In one embodiment, the composition device 100 may extract feature points while sliding a time window of a preset time length along a time axis in which brain signal data is measured.

The composition device 100 uses brain signal data (which may be a feature point; hereinafter, the brain signal data will be described on the premise that it includes the brain signal data itself or a feature point extracted from the brain signal data) as a first machine learning-based The first sound imagined by the subject for whom brain signal data was obtained can be estimated by inputting it into the classifier 520 (S120).

The first classifier 520 uses the characteristic points of the brain signal or the measurement parameters of the brain signal to match the sound imagined by the subject (may include pitch and/or beat, may be a specific note, pitch of the sound compared to the previous sound) It may include deep learning models based on CNN, etc., trained with training data labeled with (may be differences), or machine learning models such as Multi-class SVM and Quadratic Disciminant Analysis (QDA), and the types are not particularly limited. .

In one embodiment, the first classifier 520 may output probabilities for a plurality of candidate sounds based on input brain signal data. For example, each probability value for whether brain signal data measured and input from a subject imagining a specific sound at time t is a brain signal imagining multiple sounds C4, D4, and E4 can be output.

In one embodiment, the first classifier 520 may output a pitch (contour) probability of the currently imagined sound compared to the previous sound based on the input brain signal data. For example, the brain signal data measured and input from a subject who imagined a specific sound at time t has the probability that the pitch is higher (rising compared to the previous pitch), the probability that it is the same pitch, and the probability that it is the same pitch compared to the sound that was determined as the final sound at the previous time t-1. (Descent compared to the preceding sound) Each probability value can be output. Therefore, the subject can more easily compose music by imagining the note outline compared to the previous note and determining the appropriate pitch according to the progress of the music harmony using the second classifier 510 based on machine learning.

In one embodiment, the first classifier 520 may be a classifier trained to estimate sound outline, melody progression, relative scale, absolute tone, pitch type, etc. by inputting brain signal data. The outline can be of three types, such as ascending, homophonic, and descending relative to the whole tone, and the pitch can be homogeneous, sequential ascending/descending, 3rd leap ascending/descending, and 4th leap ascending/descending, etc. In the case of a relative scale, it may be a euphonious scale, an octatonic scale, a hexatonic scale, etc., starting from the tonic tone, degrees, to o'clock. Absolute sounds can be pitches that vary depending on frequency, or in the case of pitch sounds, they can be note names such as C, D, E, etc.

The composition device 100 may estimate a second sound appropriate for the current time by using a plurality of past sounds as input to the second classifier 510 based on machine learning (S130).

The second classifier 510 may be a time series-based learning model such as LSTM or RNN, or may be based on reinforcement learning. In the case of reinforcement learning, the similarity between the statistical chord progression of a specific style music score and the chord progression of the final completed score can be set as compensation.

The second classifier 510 can be trained using conventional music scores as training data. In one embodiment, the second classifier 510 may be trained using music scores classified by type of music, composer, etc. as training data. In other words, it may be a learning model trained to input a plurality of sounds and output the next sound based on scores according to a predetermined music classification.

In one embodiment, the second classifier 510 can estimate a second sound suitable for the current time based on a plurality of previously determined sounds using a learning model based on the type or composer of a specific music selected by the user. there is. Therefore, the user can compose music in the desired music style.

In one embodiment, the second classifier 510 may output each probability value of a plurality of sounds suitable for the second sound suitable for the current time.

For example, a plurality of sounds confirmed before t can be input to the second classifier 510 and the probability values for each of the plurality of sounds can be output as specific sounds suitable for time t. In other words, when the current point is t, past sounds can correspond to P(t-1), ..... P(t-n). (n >= 2)

The composition device 100 determines the final sound based on the first and second sounds estimated by the first classifier 520 and the second classifier 510 to be appropriate for time t (S140).

In one embodiment, the composition device 100 reflects weights on the probabilities according to the pitches of a plurality of first sounds estimated to be suitable for time t and the probabilities of a plurality of sounds suitable for the second sound, respectively, and then adds them up to form a plurality of sounds. The probabilities of candidate sounds may be calculated, and the sound with the highest probability among the plurality of candidate sounds may be determined as the final sound.

5, the weights (WB) are reflected in the probabilities (k1..ki) for the plurality of first sounds calculated by the first classifier 520, respectively, and the weights (WB) calculated by the second classifier 510 are reflected. After reflecting the weights (WM) on the probabilities (y1..yi) of a plurality of second notes, each of them is added for the same note, and the note with the highest value is selected as the final note (540). You can decide.

In one embodiment, the weights may be input from the user, including a weight (WB) for the probabilities (k1..ki) for the first sounds and a weight (WM) for the probabilities (y1..yi) for the second sounds. ) may be 1. Therefore, the user can further reflect the weight (WB) for the sound imagined by the user, or the weight (WM) for the sound calculated based on the conventional music score of a certain classification, to adjust the final sound in the direction desired by the user. can be decided.

In another embodiment, the weights may be set to be proportional to the accuracies of the first classifier and the second classifier. Therefore, if the performance of a specific classifier is low, the completeness of the entire song can be improved by setting it to reflect the results of other classifiers more.

In one embodiment, the probability of a plurality of candidate sounds is calculated by reflecting weights on the probability according to the pitch of the first sound and the probability of a plurality of sounds suitable for the second sound, respectively, and then summing them, and calculating the probability of a plurality of candidate sounds, and calculating the probability of a plurality of candidate sounds. A note with a high probability can be determined as the final note.

To explain with reference to Figure 6, the sounds determined before time t are D4, D4, C4, and B3 in distant chronological order, and it is assumed that the subject imagined a note suitable for time t as having a higher note contour than B3, the previous note. It is explained as follows. The probability that the first classifier 520 estimated from the brain signal measured when the subject imagined a tone with a high tone contour that the subject imagined a tone lower than before ( ), the probability of imagining the same sound ( ), the estimated probability of imagining a high note ( ) are reflected in each weight (WB), and the probabilities that the second classifier 510 is suitable for G3 and A3, which are sounds lower than B3 ( , ), the probability that the homonym B3 is suitable ( ), the probabilities that high notes C4~A5 are suitable ( ~ ) After reflecting the weight (WM) on each, the probabilities reflecting the weight for the same notes can be added to determine the highest note as the final note.

The present disclosure described above can be implemented as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is. Additionally, the computer may include a processor for each device.

Meanwhile, the program may be specially designed and configured for the present disclosure, or may be known and usable by those skilled in the art of computer software. Examples of programs may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification (particularly in the claims) of the present disclosure, the use of the term “above” and similar referential terms may refer to both the singular and the plural. In addition, when a range is described in the present disclosure, the invention includes the application of individual values within the range (unless there is a statement to the contrary), and each individual value constituting the range is described in the detailed description of the invention. It's the same.

Unless there is an explicit order or description to the contrary regarding the steps constituting the method according to the present disclosure, the steps may be performed in any suitable order. The present disclosure is not necessarily limited by the order of description of the steps above. The use of any examples or illustrative terms (e.g., etc.) in the present disclosure is merely to describe the present disclosure in detail, and unless limited by the claims, the scope of the present disclosure is limited by the examples or illustrative terms. It doesn't work. In addition, those skilled in the art will recognize that various modifications, combinations and changes may be made according to design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present disclosure should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the claims are within the scope of the spirit of the present disclosure. It will be said to belong to

This study was submitted as a result of research (Research project title: “Development of BCI-Musicing system through brain signal decoding”) conducted with the support of Samsung Future Technology Development Center.

100: Composition device
200: Brain signal measurement device
210: electrode cap
510: second classifier
520: first classifier

Claims (16)

At least a portion of each step is performed by a processor,
Receiving brain signal data as input;
Inputting the brain signal data into a first machine learning-based classifier to estimate a first sound at time t imagined by the subject at which the brain signal data was acquired;
Estimating the second sound at time t in a second machine learning-based classifier that receives a plurality of past sounds determined before time t and estimates the next sound; and
Comprising determining a final sound based on the first sound and the second sound,
BCI-based music composition method.
According to claim 1,
The step of estimating the first sound is,
Comprising the step of outputting a probability according to the pitch of the first sound by comparing it with the immediately preceding sound by the first classifier,
BCI-based music composition method.
According to claim 1,
The step of estimating the first sound is,
Comprising the step of outputting probabilities for a plurality of candidate sounds based on the brain signal data by the first classifier,
BCI-based music composition method.
According to claim 1,
The second classifier is a learning model trained to input a plurality of sounds and output the next sound based on scores according to a predetermined music classification,
BCI-based music composition method.
According to claim 1,
The step of estimating the second sound is,
Comprising the step of outputting, by the second classifier trained with preset scores, a probability of a plurality of sounds suitable for the second sound based on a plurality of past sounds,
BCI-based music composition method.
According to claim 1,
The step of determining the final sound is,
Calculating the probability of a plurality of candidate sounds by reflecting weights on the probability according to the pitch of the first sound and the probability of a plurality of sounds suitable for the second sound and adding them together; and
Comprising the step of determining the sound with the highest probability among the plurality of candidate sounds as the final sound,
BCI-based music composition method.
According to claim 1,
The step of determining the final sound is,
calculating the probabilities of a plurality of candidate sounds by adding weights to the probabilities of a plurality of sounds suitable for the first sound and the probabilities of a plurality of sounds suitable for the second sound; and
Comprising the step of determining the sound with the highest probability among the plurality of candidate sounds as the final sound,
BCI-based music composition method.
According to claim 6 or 7,
The weight is proportional to the accuracy of the first classifier and the second classifier, respectively.
BCI-based music composition method.
processor; and
A memory electrically connected to the processor and storing at least one code executed by the processor,
When the memory is executed in the processor, it inputs the brain signal data received by the processor into a first classifier based on machine learning to estimate the first sound at time t imagined by the subject at which the brain signal data was acquired, A second classifier based on machine learning that receives a plurality of past sounds confirmed before time t and estimates the next sound estimates the second sound at time t, and makes a final sound based on the first sound and the second sound. storing the code that causes the note to be determined,
BCI-based music composition device.
According to clause 9,
The memory further stores a code that causes the processor to output a probability according to the pitch of the first sound compared to the previous sound by the first classifier,
BCI-based music composition device.
According to clause 9,
the memory further stores code that causes the processor to output probabilities for a plurality of candidate sounds based on the brain signal data by the first classifier,
BCI-based music composition device.
According to clause 9,
The second classifier is a learning model trained to input a plurality of sounds and output the next sound based on scores according to a predetermined music classification,
BCI-based music composition device.
According to clause 9,
The memory further stores a code that causes the processor to output, by the second classifier trained with preset scores, a probability of a plurality of notes suitable for the second sound based on a plurality of past sounds,
BCI-based music composition device.
According to clause 9,
The memory causes the processor to calculate the probability of a plurality of candidate sounds by reflecting weights on the probability according to the pitch of the first sound and the probability of a plurality of sounds suitable for the second sound, respectively, and adding them together. further storing a code that causes the sound with the highest probability among candidate sounds to be determined as the final sound,
BCI-based music composition device.
According to clause 9,
The memory causes the processor to calculate the probability of a plurality of candidate sounds by reflecting weights on the probabilities of a plurality of sounds suitable for the first sound and the probabilities of a plurality of sounds suitable for the second sound, respectively, and then summing them up. Further storing a code that causes the sound with the highest probability among the candidate sounds to be determined as the final sound,
BCI-based music composition device.
The method of claim 14 or 15,
The weight is proportional to the accuracy of the first classifier and the second classifier, respectively.
BCI-based music composition device.

Images