KR102497878B1 - Vocal transcription learning method and apparatus for performing learning based on note-level audio data - Google Patents

Abstract

A vocal transcription device for performing learning based on note-level audio data according to an embodiment comprises: one or more processors; and a memory module for storing instructions executable in the one or more processors. The processors include: a first artificial neural network having first audio data in a frequency region as first input information and outputting, as first output information, pitch information including the pitch information of a vocal for each frame of the first audio data; a preprocessing module for converting the first output information into first learning data including vocal information for each note; a third artificial neural network having second audio data in a frequency region as third input information and outputting, as third output information, pitch information including the pitch information of a vocal for each note of the second audio data; and a postprocessing module for converting the third output information into vocal information for each note, wherein the third artificial neural network can perform learning for the third artificial neural network based on the first learning data. Therefore, provided is a technology based on deep learning, wherein notes corresponding to a vocal melody in polyphonic music can be predicted more accurately.

Description

Vocal transcription learning method and apparatus for performing learning based on note-level audio data}

The present invention relates to a vocal transcription learning method and apparatus for learning based on note-level audio data, and more particularly, a technique for accurately outputting pitch information for vocals for input audio using an artificial neural network. And an invention related to a technology for efficiently learning an artificial neural network using the same.

Chasebo means to transfer music that is not originally notated into sheet music, and the purpose of transcription is divided into cases where it is used after being arranged and used in a repertoire or for academic analysis. There is a prescriptive writing style and a descriptive writing style that meticulously writes down as it is performed.

On the other hand, automatically transcribing the music of the input sound source into sheet music by a computer processor or the like is called automatic transcription. It means recognizing the height (pitch, interval), length (field, duration), and lyrics of the song and displaying the result in the form of a score. In the case of automatic transcription, the system automatically recognizes and analyzes the musical characteristics of the singer's song, compared to the existing method in which an expert familiar with music listens to and transcribes the song himself, so that the general public can easily record it. There are advantages to using it.

As for the automatic transcription method known so far, a method of combining segmented intervals in phoneme units so that feature information extracted from a continuous voice signal can be recognized as each note and using it as syllable boundary information, and a method of using pitch intervals in a voice signal There is a method of forming a syllable segment using energy information obtained by connecting the maximum value of the voice generated each time.

However, in the conventional automatic transcription system as described above, when dividing syllables due to the continuous characteristics of the voice signal, there is a problem in that the efficiency is significantly lowered in the part where the boundary is ambiguous, and the pitch is recognized for each section of the predicted syllable boundary. For this, there is an inconvenience of adding a process of finding a representative value of pitch information. In addition, it has a problem that it cannot provide satisfactory results because it is impossible to detect bars and thus cannot show the recognition result of a song in the form of a complete sheet music. There was a downside that it was possible.

In addition, since people usually sing according to their own style, individual differences are very large because each song has a different speed and time, but conventional methods for sound field recognition are based on standardized standard data. Since a mapping method is used, there is a disadvantage in that each person cannot adapt to the different song input speed.

Korean Patent Publication No. 10-2015-0084133 (published on July 22, 2015) - 'pitch recognition using sound interference and a method for transcribing scales using the same' Korean Patent Registration No. 10-1696555 (2019.06.05.) - 'Text location search system through voice recognition in video or geographic information

Therefore, a vocal transcription learning method and apparatus for learning based on note-level audio data according to an embodiment is an invention designed to solve the described problem, and transcription transcription containing pitch information and vocal information more effectively than the prior art. Its purpose is to provide a deep learning-based technology that more accurately predicts notes corresponding to vocal melodies in polyphonic music by outputting information.

More specifically, a vocal transcription learning method and apparatus for learning based on note-level audio data according to an embodiment uses a model that converts frame-level transcription information into note-level transcription information, An object of the present invention is to provide a method and apparatus for effectively outputting note-level transcription information with only a small amount of frame-level data.

A vocal transcription apparatus for performing learning based on note-level audio data according to an embodiment includes one or more processors and a memory module storing instructions executable by the one or more processors, wherein the processor comprises: A first artificial neural network that takes audio data as first input information and outputs pitch information including vocal pitch information in frames for the first audio data as first output information; A pre-processing module for converting first output information into first training data including vocal information in units of notes, second audio data in the frequency domain as third input information, and notes (notes) for the second audio data A third artificial neural network that outputs pitch information including vocal pitch information in units of notes as third output information and a post-processing module that transforms the third output information into vocal information in units of notes, The third artificial neural network may perform learning on the third artificial neural network based on the first learning data.

The pre-processing module and the post-processing module may convert frame-unit pitch information into note-level pitch information using a pitch quantization method and a rhythm quantization method.

The vocal transcription device for performing learning based on the note-level audio data includes a first memory module for storing the first learning data and second learning data including pitch information labeled at the note level. A second memory module may be included.

The third artificial neural network may perform learning on the third artificial neural network based on the second learning data.

A vocal transcription apparatus for performing learning based on note-level audio data according to an embodiment includes one or more processors and a memory module storing instructions executable by the one or more processors, wherein the processor comprises: A first artificial neural network that takes audio data as first input information and outputs pitch information including vocal pitch information in frames for the first audio data as first output information; A pre-processing module for converting first output information into first training data including vocal information in units of notes, second audio data in the frequency domain as third input information, and notes (notes) for the second audio data A third artificial neural network that outputs pitch information including pitch information of vocals in units of note) as third output information, and uses third audio data in the frequency domain as fifth input information, and the third audio data A fifth artificial neural network outputting pitch information including vocal pitch information in units of notes as fifth output information and second learning data including vocal information in units of notes as the third output information and a second post-processing module for converting to , wherein the third artificial neural network may perform learning based on at least one of the second learning data and the third learning data.

The vocal transcription apparatus for learning based on the note-level audio data may further include a second post-processing module for transforming the fifth output information into note-level vocal information.

In the method of learning a note-level automatic vocal transcription device using a processor including an artificial neural network according to an embodiment, the processor sets first audio data in the frequency domain as first input information, and the first audio data a first output information outputting step of outputting the first output information using a first artificial neural network that outputs pitch information including pitch information of vocals as first output information in units of frames for the first output information; A pre-processing step of converting, by a processor, the first output information into first training data including vocal information in units of notes; the processor taking second audio data in the frequency domain as third input information; Third output information output that outputs the third output information using a third artificial neural network that outputs pitch information including vocal pitch information in units of notes for audio data as third output information a post-processing step in which the processor converts the third output information into vocal information in units of notes, and a learning step in which the processor performs learning for the third artificial neural network based on the first training data. can include

A note-level automatic vocal transcription method and apparatus using an artificial neural network according to an embodiment uses the estimated vocal information in inferring pitch information including vocal information, and utilizes the estimated pitch information in inferring the vocal information. Therefore, there is an advantage of being able to infer pitch information and vocal information included in audio data more effectively than the prior art. Accordingly, there is no pre-processing step for separating vocals in transcription, so the amount of calculation is greatly reduced, and there is an advantage that vocal transcription can be performed with high accuracy even without separating vocals.

In addition, a method and apparatus for automatic vocal transcription of a note level using an artificial neural network according to an embodiment converts frame-level pitch information into pseudo-note level pitch information, and then performs note-level transcription of the converted data. Since it is used as training data of an artificial neural network, there is an advantage in that an artificial neural network capable of efficiently outputting note-level pitch information can be trained with less note-level training data.

In addition, since learning based on semi-supervised learning is performed in learning the artificial neural network, it is possible to efficiently learn the artificial neural network that outputs note-level pitch information with only a small amount of unlabeled data.

Accordingly, the note-level automatic vocal transcription method and apparatus using an artificial neural network according to an embodiment can perform flexible and accurate transcription for sound sources of various genres, and can directly generate vocal melodies from polyphonic music without a source separation preprocessing process. Since it is possible to transcribe, there is an advantage that the transcribing speed can be greatly improved compared to the prior art.

1 is a block diagram of a music transcription system including a music transcription device according to an embodiment of the present invention.
2 is a block diagram showing some components of a music transcription device according to an embodiment.
3 is a diagram illustrating input information and output information of a first artificial neural network according to an embodiment.
4 is a block diagram illustrating some components constituting a first artificial neural network according to an embodiment.
5 is a block diagram illustrating some components of a RAZNET block according to an exemplary embodiment.
6 is a diagram showing input information and output information of a second artificial neural network according to an embodiment of the present invention.
7 is a diagram showing the configuration of a first artificial neural network module including a first artificial neural network and a second artificial neural network according to an embodiment of the present invention.
8 is a diagram showing a first artificial neural network module including a first artificial neural network and a second artificial neural network according to another embodiment of the present invention.
9 is a diagram showing a method of calculating an integrated loss function of a music transcription artificial neural network module according to the present invention.
10 is a diagram illustrating components of an automatic music transcription apparatus using an artificial neural network according to an embodiment.
11 is a diagram for explaining a pre-processing module of an automatic music transcription apparatus according to an embodiment.
12 is a diagram illustrating components of an automatic music transcription device using an artificial neural network according to another embodiment.

Hereinafter, embodiments according to the present invention will be described with reference to the accompanying drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are displayed on different drawings. In addition, in describing an embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function hinders understanding of the embodiment of the present invention, the detailed description thereof will be omitted. In addition, embodiments of the present invention will be described below, but the technical idea of the present invention is not limited or limited thereto and can be modified and implemented in various ways by those skilled in the art.

In addition, terms used in this specification are used to describe embodiments, and are not intended to limit and/or limit the disclosed invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, terms such as "include", "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or the existence or addition of more other features, numbers, steps, operations, components, parts, or combinations thereof is not excluded in advance.

In addition, throughout the specification, when a part is said to be “connected” to another part, this is not only the case where it is “directly connected”, but also the case where it is “indirectly connected” with another element in the middle. Terms including ordinal numbers, such as "first" and "second" used herein, may be used to describe various components, but the components are not limited by the terms.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted.

Meanwhile, the title of the present invention is described as 'a note-level automatic vocal transcription method and apparatus using an artificial neural network', but for convenience of description below, 'a note-level automatic vocal transcription method and apparatus using an artificial neural network' is 'music transcription'. It will be abbreviated as 'device'.

1 is a block diagram of a music transcription system including a music transcription device according to an embodiment of the present invention.

Referring to FIG. 1, a music transcription system 1 may include a user terminal 100 that provides an automatic transcription service and a music transcription device 200 that performs an automatic transcription task, and the user terminal 100 moves It is implemented as a terminal, and the music transcription device 200 may be implemented as a remote server.

Accordingly, the user terminal 100 is a personal communication system (PCS), a global system for mobile communication (GSM), a personal digital assistant (PDA), an international mobile telecommunication (IMT)-2000, a code division multiple access (CDMA)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, smart watch, smart glass, wearable It may include all kinds of handheld-based wireless communication devices such as wearable devices.

The server on which the music transcription device 200 is implemented means a typical server. A server is computer hardware on which a program is executed, and monitors or controls the entire network, such as printer control or file management, or a mainframe. or connection to another network over a public network, sharing software resources such as data, programs, files, modems, fax machines, and printers. It can support sharing of hardware resources such as other equipment.

The user terminal 100 may include a recording unit 110, a communication unit 120, a MIDI generation unit 130, and a display unit 140.

The recording unit 110 may record a sound source. In detail, the recording unit 100 may record a sound source heard from outside the user terminal 100 or a sound source being reproduced in the user terminal 100 itself. The sound source recorded by the recording unit 100 may be transmitted to the music transcription device 200 through the communication unit 120 .

The communication unit 120 may transmit data including a sound source recorded by the music transcription device 200 or receive data from the music transcription device 200 . For example, the communication unit 120 may receive the transcribed sheet music from the music transcription device 200 or may receive additional service information related to the transcribed sheet music.

The MIDI generating unit 130 may generate and reproduce a Musical Instrument Digital Interface (MIDI) file according to the transcribed sheet music.

The display unit 140 may externally output the transcribed sheet music or other additional service information so that the user can recognize it. Accordingly, the display unit 140 may include various display panels such as a liquid crystal display (LCD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel. there is. Meanwhile, when the display unit includes a graphical user interface (GUI) such as a touch pad, that is, a software device, it may serve as an input unit (not shown) that receives a user's input.

The music transcription device 200 may include a processor 300, a memory module 400, an additional service module 500, and a user management module 600.

The processor 300 may automatically generate transcription for a sound source stored in the memory module 400 or a sound source received from the user terminal 100 using an artificial neural network module. A detailed description of this will be described later.

The memory module 400 may store a sound source received from the user terminal 100 or store various data necessary for the processor 300 to learn and infer the artificial neural network module. Reference data received through the user terminal 100 or the like or various data generated by the processor during learning or reasoning may be stored.

The additional service module 500 may generate additional service information related to the score transcribed by the processor 300 .

For example, the additional service module 500 may provide a composition assistant service as an additional service. The composer helper service provided according to the present invention is one of music application services using the user terminal 100, and the sound source collected from the user terminal 100 is automatically recorded using the music transcription device 200 implemented as a remote server. It may refer to a composition helper service with improved performance that converts music into music and transmits it to the user terminal 100 for storage.

Here, the composition helper service with improved performance means that the music transcription device 200 that provides more accurate transcription than the prior art is utilized.

Even a person who is not familiar with music can have a melody that comes to mind through humming, and may have a desire to complete a composition by connecting this melody. For example, suppose a mother sings a lullaby for her baby. After recording this humming lullaby, it can be stored in the form of a sound source and then played back. The song will be able to be reproduced by anyone next time.

In order to receive such a composition assistant service, a user records a sound source through the mobile terminal 100 and transmits it to the music transcription device 200 implemented as a remote server. In the mobile terminal 100, sound recording may be performed through the recording unit 110 and transmission of the sound source may be performed through the communication unit 120.

In addition, the additional service module 500 may also provide a melody-based music search system and a melody-based similar sound source search service based on data generated by the processor 300 .

In the case of a registered user, the music transcription device 200 may receive a sound source file, analyze and transcribe the sound source, and transmit the transcribed result to the user terminal 100 . In the music transcription device 200, sound source analysis and transcription are performed through the processor 300, and the transcribed result may be transmitted to the user terminal 100 through a communication unit.

The user terminal 100 can display the transcribed score through the MIDI generator 130 and the display unit 140 and reproduce music according to the score, and the user sees the transcribed score and listens to the music according to the score. You can complete your composition by adding, deleting, modifying, etc.

The user management module 600 may store data for managing users of the user terminal 100 using the music transcription device 200 . Meanwhile, although not shown in the drawings, the music transcription device 200 may include a communication unit, and the communication unit may transmit data to the user terminal 100 . For example, the communication unit may transmit musical score information transcribed by the processor 300 to the user terminal 100 or may transmit various additional service information generated by the additional service module 500 .

2 is a block diagram showing some components of a music transcription device according to an embodiment, and FIG. 3 is a diagram showing input information and output information of a first artificial neural network according to an embodiment. 4 is a block diagram showing some components constituting a first artificial neural network according to an embodiment, and FIG. 5 is a block diagram showing some components of a RAZNET block according to an embodiment.

Referring to FIG. 2, the music transcription device 200 according to an embodiment may include a processor 300, a memory module 400, an additional service module 500, and a user management module 600, and a processor ( 300) may include a first artificial neural network 311 including a first artificial neural network 311 and a second artificial neural network 312. Descriptions overlapping those described in FIG. 1 will be omitted, and the music transcription artificial neural network module will be described in detail in FIGS. 2 to 7 .

Meanwhile, in FIG. 1, the processor 300 is illustrated as including only the first artificial neural network 311, but the processor 300 includes a third artificial neural network 313 and a fourth artificial neural network 314, and the second artificial neural network 314 is included. It may include a neural network 320 and a third artificial neural network 330 including a fifth artificial neural network 115 and a sixth artificial neural network 316, and each artificial neural network module is a music transcription artificial intelligence module in view of its nature. It may also be called a neural network module.

The first artificial neural network 311 takes the first input information 10 including audio data as input information and outputs pitch information in units of frames for vocals included in the input audio data. It means a neural network module, and the second artificial neural network 312 takes the second input information 20 obtained by summing the intermediate information of the first artificial neural network 311 as input information, It refers to an artificial neural network module that outputs vocal information, which is information about existence or nonexistence.

The pitch information output by the first artificial neural network 311 includes information on the presence or absence of a vocalist in each frame for each frame of input audio data and, if a vocalist exists, pitch information of the vocalist.

Referring to FIGS. 3 and 4, the first artificial neural network 311 will be described in detail. The first artificial neural network 311 according to an embodiment converts first input information 10 including audio data into input information. , and means a pre-learned artificial neural network module whose output information is the first output information 20 including pitch information for vocal information included in the audio data, and the first artificial neural network 311 An inference session (not shown) for inferring the first output information 20 based on the first input information 10, the first input information 10, the first output information 20, and the first reference data A learning session (not shown) may be included to adjust and update the parameters of the first artificial neural network 311 in a way to increase the accuracy of the first output information 20 based on the basis.

The first input information 10 means sound source data for which the user wants to generate transcription information, and the first input information 10 may mean data generated based on a frame. In FIG. 4, as an example, the first input information 10 input to the first artificial neural network 311 is shown as data inputting 31 frame data at a time, but the embodiment of the present invention is not limited to this example, and the first input information 10 The number of frames for the audio data included in the first input information 10 input to the artificial neural network 311 may be set to various numbers.

The first output information 20 may include pitch information for each frame of audio data input to the first artificial neural network 311 . Therefore, the first output information 20 may include information about the presence or absence of a vocalist and the size of a vocalist for each frame, and when no vocalist exists, the output information for the corresponding frame may be output as 0. .

Therefore, information for each frame output by the first artificial neural network 311 is output as NV (Non Vocal) information in a frame where no vocal exists at all, as shown in the lower right of FIG. 3, and in a frame where vocal exists. Information is output as pitch information (eg, D2, B5, etc.) information in each frame.

Meanwhile, according to the designation name, the first artificial neural network 311 may be referred to as a pitch information output artificial neural network module, and in FIG. 4 , for example, the first input information 10 input to the first artificial neural network 311 is 31 Since it is shown that two frame data are input, the information output from the first artificial neural network 311 may include pitch information for each of the 31 frames. Therefore, as shown in FIG. 4, the output information is For each frame (first frame, second frame, third frame to 31st frame), it may include information on whether or not there is a sound level and, if so, how loud it is. there is.

The first artificial neural network 311 may be implemented by combining several convolution blocks and resnet blocks constituting a known artificial neural network. As an example, as shown in FIG. 4, the first artificial neural network 311 includes a convolution block 121, a first Resnet block 122, a second Resnet block 123, and a third Resnet block 124. , a polling block 125 and a 1LSTM block 126, and the like.

Each component of the first artificial neural network 311 shown in FIG. 4 is merely an example, and the embodiment of the present invention is not limited to the components shown in FIG. 4 . Therefore, for example, not one but a plurality of convolution blocks may be provided, and one, two, or four or more blocks instead of three Resnet blocks may be provided.

Looking in detail at the blocks constituting the first artificial neural network 311 according to the present invention, the convolution block 121 input to the first artificial neural network 311 is the input first input information 10 It means a block that performs a convolution operation on , and a convolution operation typically used in a CNN network can be performed.

A ResNet block means a block that performs Residual Network (ResNet). The Resnet block shown in FIG. 4 means a ResNet implemented by modifying the known ResNet according to the purpose of the present invention. In one embodiment, a plurality of Resnet blocks may be connected in series. Therefore, the output information of the previous Resnet block can be input as the input information of the serially connected Resnet block.

The first artificial neural network 311 according to the present invention, as shown in FIG. Experimental results show that including the Resnet block outputs the best output information when considering the accuracy of the output information and the efficiency of the process, so that the first artificial neural network 311 includes three Resnet blocks. Although illustrated, the embodiment of the present invention is not limited thereto, and the number and arrangement of the RAZnet blocks may be variously modified according to the purpose of the invention.

Each Resnet block of the first artificial neural network 311 may be implemented as a modified residual network as shown in FIG. 5 . Accordingly, the first Resnet block 112 may receive the output information of the convolution block 121 as input information and output the 1-1 intermediate information 11 as output information through various network operations.

Specifically, the second Resnet block 113 receives the input information of the 1-1 intermediate information 11, which is the output information of the second Resnet block 112, and converts it to the 1-1 intermediate information 11. After performing several network operations on the 2-1st intermediate information 12 can be output as output information. The third RESNET block 114 may also receive the 2-1 intermediate information 12 as input information and output the 3-1 intermediate information 13 as output information by the same processor.

Referring to FIG. 5, the first Resnet block 112 will be described (the second Resnet block and the third Resnet block also have the same structure as the first Resnet block). The first Resnet block 112 As shown in FIG. 5, sequentially includes a BN/LReLU block 121, a MaxPool (1X4) block 122, a Conv 2D block 123, a BN/LReLU block 124, and a Conv 2D block 125. and each block can perform an operation corresponding to the name of the block. Since calculations such as BN/LReLU, MaxPool, and Conv 2D correspond to known calculation technologies, a detailed description thereof will be omitted.

The difference between the first Resnet block 112 according to the present invention and the known Resnet network is explained, the information output from the first Resnet block 112 is the information output from the Conv 2D block 125 and information output from the MaxPool (1X4) block 122 may be summed and output as the 1-1st intermediate information 11. That is, if the information output from the MaxPool (1X4) block is X information, X information becomes Y information through convolution 2D operation in the Conv 2D block, so the first intermediate information 11 is finally Y information and Conv 2D It may be implemented as summation information of Z information output in block 325 .

The third intermediate information 13 output from the third RESnet block 114 may be input to the first LSTM block 316 through the pulling block 115 . The first LSTM block means a block implemented as a long short-term memory (LSTM) network, which is a kind of RNN network. In the case of audio data, since the input audio has a continuous characteristic, using an LSTM network that utilizes the result of previous data has the advantage of effectively outputting pitch information or pitch information for the input audio data. In the case of the first artificial neural network 310 shown in FIG. 4, since audio data corresponding to 31 frames is used as the first input information 10, the 1st LSTM block 316 also utilizes a network composed of 31 layers can be implemented

The first output information 20, which is information output according to the 1LSTM block 316, may include pitch information of the input audio data.

Specifically, the first output information 20 may output pitch information of a voice corresponding to each input frame, and the first output information 20 may determine whether a human vocal exists in the audio data or not 1-1st output information (Omv), which is data information about whether or not there is a voice, and 1-2nd output information (Om) including information on whether vocals exist and, if vocals exist, 1-2nd output information (Om) including pitch information of the sound can do.

For example, as shown in FIG. 4 , when vocals exist in the first frame and the second frame and vocals do not exist in the third frame, the 1-1 output information Omv is the first frame and the second frame. has ON information, and has OFF information in the third frame. Therefore, in this case, 1-1 output information is output as NV (Non Vocal) information in the third frame and V (Vocal) information in the first and second frames.

On the other hand, as described above, the first-second output information (Om), which is information about the pitch of the sound, numerically calculates the ON/OFF information of the vocal and the information about the pitch of the sound in each frame if the vocal is present. information may be included. Accordingly, in this case, since information corresponding to the pitch of the vocal in the first frame and the second frame and no vocal in the third frame, information may be output as 0.

So far, the first artificial neural network 311 included in the first artificial neural network module 310 has been studied in detail. Hereinafter, the second artificial neural network 312 included in the first artificial neural network module 310 and connected in parallel with the first artificial neural network 311 will be described in detail.

6 is a diagram showing input information and output information of a second artificial neural network according to an embodiment of the present invention, and FIG. 7 includes a first artificial neural network and a second artificial neural network according to an embodiment of the present invention. It is a diagram showing the configuration of the first artificial neural network module.

Referring to FIGS. 6 and 7, the second artificial neural network 312 will be described in detail. The second artificial neural network 312 according to an embodiment sums various intermediate information output from the first artificial neural network 311. Second output information including vocal information on whether or not a vocal exists for each frame in the audio data included in the first input information 30 as input information ( 40) as output information, and means a pre-learned artificial neural network module.

Therefore, although not shown in the figure, the second artificial neural network 312 includes an inference session (not shown) for inferring the second output information 40 based on the second input information 30, and the second input information 30 and a learning session (not shown) for adjusting and updating the parameters of the second artificial neural network 312 in a way to increase the accuracy of the second output information 40 based on the second output information 40 and the second reference data. can include

Looking at the structure of the second artificial neural network 312 in detail, as shown in FIG. 7, the second artificial neural network 312 includes a convolution block 121 receiving second input information 30 and A second LSTM block 122 outputting the output information 40 may be included.

Specifically, the second input information 30 may be information obtained by combining intermediate information output from various blocks constituting the first artificial neural network 311 as input information. Specifically, the first Resnet block 112 The first intermediate information 11 output from , the second intermediate information 12 output from the second Resnet block 113, the third intermediate information 13 output from the third Resnet block 114, and polling It may be implemented as summation information of the fourth intermediate information 14 output in block 115 .

In another embodiment, the second input information 30 may be implemented as information that has passed through a plurality of max polling blocks. Specifically, the second input information 30 is the 1-1 intermediate information 11 The 1-2 information 21 generated through the 1 MP block 131 as a polling block and the 2-1 intermediate information 12 generated through the 2 MP block 132 as a second max polling block The 2-2 information 22 and the 3-1 intermediate information 13 pass through the 3-2 intermediate information 23 and the polling block 115 through the 3 MP block 133, which is the third max polling block. It can be composed of the sum of the fourth intermediate information (14). The second input information 30 configured in this way may be input to the second LSTM block 122 through the convolution block 121 and finally output as the second output information 40 .

6 shows that the second artificial neural network 312 includes only the convolution block 121 and the second LSTM block 122 for convenience of description, but according to an embodiment of the present invention, the second artificial neural network 312 It may include a plurality of MP blocks shown in FIG. 6 , and the number of MP blocks may be implemented as a number corresponding to the number of Resnet blocks included in the first artificial neural network 311 .

The second output information 40, which is information output according to the second LSTM block 122, may include vocal information about the input audio data.

Specifically, the second output information 40 is information (V) on whether or not vocal information exists in each frame based on the audio data included in the first input information 10 or a discussion on whether or not vocal information exists. It may include vocal information (NV).

The second output information 40 output by the second artificial neural network 312 corresponds to information substantially included in the first output information output by the first artificial neural network 311 . However, in the case of the first artificial neural network 311, if the focus is on outputting pitch information, which is the height information of vocals, in audio data, in the case of the second artificial neural network 312, the section and presence of vocals in audio data It can be understood as an artificial neural network focused on outputting the presence or absence of information about the section that does not. Accordingly, the second artificial neural network 312 may also be referred to as a vocal information output artificial neural network according to characteristics of information outputting vocal information.

8 is a diagram showing a first artificial neural network module including a first artificial neural network and a second artificial neural network according to another embodiment of the present invention.

In the case of the first artificial neural network module according to FIG. 8, the basic configuration is the same as the configuration described in FIG. 7, but there is a difference in the configuration of the first artificial neural network 311.

Specifically, the first artificial neural network 311 according to FIG. 8, as shown in the figure, outputs the information output from the first LSTM block 116 and the second LSTM block ( 122) may include a summing block 317 for outputting final information of the first artificial neural network 311 by taking the summed information as input information. The summing block 317 is composed of a dense layer and an FC layer, and after summing the information output from the 1st LSTM block 316 and the information output from the 2nd LSTM block 122, respectively, the summed information By performing a network operation based on , it is possible to output the first output information 20 as output information.

In addition, although not shown in the drawing, the information output from the summation block 317 may be input again as information of the second artificial neural network, and the second artificial neural network 312 may be used to output vocal information. Specifically, the second artificial neural network 312 further includes a summation block serially connected to the 2LSTM block 122, so that the information output from the 2LSTM block 122 and the summation block 317 After summing up the information, a network operation may be performed based on the summed information to output the second output information 40 as final output information.

In the case of having the configuration shown in FIG. 8, since the vocal information output from the second artificial neural network 312 can be used as intermediate input information once again in the first artificial neural network 311 outputting pitch information, the input Vocal information can be clearly distinguished from the audio data to be used. Accordingly, there is an advantage in that pitch information for vocals can be output more clearly by using such information.

In addition, since the pitch information output from the first artificial neural network 311 can be utilized as intermediate input information once again in the second artificial neural network 312 that outputs vocal information, vocal information can be more accurately recorded in the input audio data. There are distinct advantages.

So far, the specific configuration and process of the first artificial neural network module corresponding to the artificial neural network module for music transcription according to the present invention have been studied. Hereinafter, the learning method of the first artificial neural network module will be described.

The first artificial neural network 311 and the second artificial neural network 312 according to the present invention may perform learning based on each artificial neural network module in inferring pitch information and vocal information, respectively. That is, the first artificial neural network 311 may calculate a loss function based only on pitch information, and then perform learning in a direction of increasing the accuracy of the pitch information using the first reference data, and the second artificial neural network 312 ) can calculate a loss function based only on vocal information, and then perform learning in the direction of increasing the accuracy of pitch information using the second reference data.

In addition, the first artificial neural network module 310 according to the present invention does not independently learn the first artificial neural network 311 and the second artificial neural network 312, but the first artificial neural network 311 and the second artificial neural network 311. After summing the output information of the neural network 312, the learning may be performed by calculating a loss function based on the summation. This will be examined in detail through FIG. 9 .

9 is a diagram showing a method of calculating an integrated loss function of a music transcription artificial neural network module according to the present invention.

Referring to FIG. 9 , the first loss function (Lpitch) of the first artificial neural network 311 according to the present invention may be configured based only on information output from the first artificial neural network 311 . That is, as shown on the left side of FIG. 9, after generating information (a1) on whether pitch information is present or information (a2) on whether there is no pitch information for each frame, the generated information and the first reference information A first loss function (Lpitch) is generated based on the difference.

The second loss function LVocal of the second artificial neural network 312 may be generated based on vocal information and melody information. That is, as shown on the right side of FIG. 9, among the components constituting the first loss function, the first vocal information (V1), which is information about the section (a1) with pitch information, and the section without pitch information (actually no vocal) After combining the first non-vocal information (NV1), which is information about the section a2), the second vocal information (V2), which is information about the section in which vocal information exists, among the information output from the second artificial neural network 311 A second loss function (LVocal) may be generated by summing both LV and second non-vocal information (NV2), which is information about a section in which no vocal information exists.

Then, the loss function of the entire artificial neural network module can be expressed as the sum of the first loss function (Lpitch) and the second loss function (LVocal), which is the total loss function Ltotal. can

Equation (1) - Ltotal = Lpitch * (a*LVocal)

In Equation (1), a means a coefficient, and in the case of the present invention, 0.1, 0.5 or 1 may be applied.

Since the loss function thus generated adjusts parameters of the entire artificial neural network in consideration of both vocal and pitch information, there is an advantage in that vocal and pitch information can be output more accurately.

So far, we have looked into the first artificial neural network module that outputs vocal information and pitch information in detail. Hereinafter, as another embodiment of the present invention, a second artificial neural network module that outputs note-level transcript information by utilizing output information of the first artificial neural network module will be described in detail.

10 is a diagram showing components of an automatic music transcription device using an artificial neural network according to an embodiment, and FIG. 11 is a diagram for explaining a preprocessing module of the automatic music transcription device according to an embodiment.

Referring to FIG. 10, the automatic music transcription apparatus according to an embodiment includes a first artificial neural network module 310 outputting pitch information of audio data in frame-level units, and a pitch output by the first artificial neural network module 310. A pre-processing module 340 for generating first learning data by converting data into note-level pitch information for the audio data based on the information, a first memory module 210 for storing the first training data, and note level The second memory module 220 for storing the second training data labeled with unit pitch information, learning is performed based on at least one of the first training data and the second training data, and input frame level unit The second artificial neural network module 320 outputs pitch information in frame level units for the audio data of , and the post-processing module 350 transforms the frame level unit information output from the second artificial neural network module into note level unit information. ) may be included.

In the automatic music transcription device according to FIG. 10, the first artificial neural network module 310 and the second artificial neural network module 320 have almost the same configuration as the artificial neural network module described in the previous figure, but as one embodiment, the first artificial neural network module 310 The artificial neural network module 310 may borrow the artificial neural network module according to FIG. 7, and the second artificial neural network module 320 may borrow the artificial neural network module according to FIG. 8. In this case, the first artificial neural network module 310 plays a role of an artificial neural network generating learning data necessary for the second artificial neural network module 320 to learn, and the second artificial neural network module 320 may actually play a role of automatic vocal transcription.

If overlapping descriptions of the first artificial neural network module 310 and the second artificial neural network module 320 are omitted, and other components and overall processes will be described, the preprocessing module 340 is the first artificial neural network module 310 ) may process pitch information in units of frames output by converting pseudo-note level pitch information.

Specifically, as shown in FIG. 10, the preprocessing module 340 converts pitch information in units of frames into pseudo-note level pitch information through largely two pitch quantization processes and rhythm quantization processes. convert The reason why it is called pseudo-note level rather than note level is that the data is not accurately collected at the note level from the beginning, but the pitch information in units of frames is processed and transformed into the note level, so there is a difference from the exact note level data. Therefore, it is called pseudo-note level.

The pitch quantization process performed by the preprocessing module 340 refers to a pitch quantization process that rounds continuous pitches to semitone steps. Specifically, in many pitch estimation models based on artificial neural networks, the output is expressed as a softmax function for quantized pitch values, where adjacent pitches tend to be much smaller than semitones. Therefore, the pitch quantization process is to take the most reliable pitch and quantize it to the nearest MIDI note number.

When the pitch quantization process is completed, the preprocessing module 340 performs a rhythm quantization process as a next step. The rhythm quantization process refers to a process of snapping pieces of the quantized pitch line in beat-based units.

For example, the process of flattening the quantized pitch with three intermediate filters is performed, and the size of the median filter is set to 1/32, 1/16, and 1/12 beats respectively at a given tempo to make the filtered output in a beat-based unit. can The number of filters may be used differently depending on the usage environment, but it was found experimentally that the three cascaded filters improved the quality of the label the most.

When the rhythm quantization process is completed, the pre-processing module 340 removes small pieces that are short to view as a song, and finally sets a simple rule to minimize octave errors to finish the pre-processing process. It is stored in the memory module 210 and can be used as learning data when the second artificial neural network module learns.

Data stored in the second memory module 220 refers to data including pitch information labeled as a note-level.

The second artificial neural network module 320 takes second audio data in the frequency domain as third input information, and third outputs pitch information including vocal pitch information in frame units for the second audio data. The third artificial neural network output as information 60 and the intermediate output information output from the blocks constituting the third artificial neural network are used as fourth input information, and for the second audio data, in units of frames, vocal It may include a fourth artificial neural network that outputs vocal information about the presence or absence of as fourth output information.

The third artificial neural network and the fourth artificial neural network of the second artificial neural network module 320 correspond to the first artificial neural network 311 and the second artificial neural network 312 of the first artificial neural network module 310, respectively. Correspondingly, most of the components are the same. However, as mentioned above, as one embodiment, the first artificial neural network module 310 borrows the artificial neural network module according to FIG. 7, and the second artificial neural network module 320 borrows the artificial neural network module according to FIG. 8. It can be.

Describing the learning method of the second artificial neural network module 320, the second artificial neural network module 320 uses the first learning data stored in the first memory module 210 as learning data and the second memory module 320. ) Learning may be performed based on at least one of the second learning data stored in ).

In general, in vocal transcription, note-level transcription, which outputs transcription information with onset, offset, and note information, is more accurate than frame-level transcription in the frequency domain. I can pass on the information of the chaebol.

However, due to the nature of the song, each person sings the same note differently in the note-level transcription, and since there are various variables, it is very difficult to accurately predict the note using a heuristic method. Crucially, in order to proceed with learning using an artificial neural network, there must be data labeled at the note level, but there is very little data labeled at the level of a score that accurately matches the audio, making it difficult to perform learning using an artificial neural network. has a very difficult problem.

However, the automatic vocal transcription apparatus according to the present invention utilizes the first artificial neural network module 310 and the pre-processing module 340 to convert existing frame-based pitch information to a pseudo-note level having a similar character to an actual note level unit. Since it can be easily converted into pitch information of , and the second artificial neural network module 320 can perform learning based on the data generated in this way, the output data is data that has characteristics very similar to note-level data. can be generated, so there is an advantage of increasing the efficiency of the note-level automatic transcribing device.

That is, the model of the second artificial neural network module 320 itself predicts the result at the frame level because its components are very similar to the configuration of the first artificial neural network module 320, but in performing learning, the similar-note level Learning was performed with the first learning data and the second learning data of the note level, and since time series learning is performed together at the LSTM stage that outputs the output information, the final output information has a very similar nature to the note level information, In fact, the effect of outputting transcription information at the note level can be obtained.

The post-processing module 350 may play a role of converting frame-level output information output from the second artificial neural network module 320 into note-level transcription information. In fact, the post-processing module 350 converts frame-based pitch information into note-level pitch information through the pitch quantization process and rhythm quantization process performed by the pre-processing module 340 described above, can be printed out. Since the processor for this has been described in detail above, it will be omitted below.

12 is a diagram illustrating components of an automatic music transcription device using an artificial neural network according to another embodiment.

Referring to FIG. 12, the automatic music transcription apparatus according to an embodiment includes a first artificial neural network module 310 outputting pitch information of audio data in frame level units, and a pitch output by the first artificial neural network module 310. A pre-processing module 340 for generating first learning data by converting data into note-level pitch information for the audio data based on the information, a first memory module 210 for storing the first training data, and note level The second memory module 220 for storing the second training data labeled with unit pitch information, learning is performed based on at least one of the first training data and the second training data, and input frame level unit The second artificial neural network module 320 outputs pitch information in unit of note level for the audio data of , and the first post-processing module transforms the information in unit of frame level output from the second artificial neural network module into the information in unit of note level. (351), the third memory module 230 in which third learning data, which is random noise learning data, is stored, information output from the first post-processing module 351, and at least one of the third learning data. The third artificial neural network module 330 that performs learning and outputs pitch information in note level units for the input audio data in frame level units and the information in frame level units output from the third artificial neural network module 330 A second post-processing module 352 for modifying note-level information may be included.

The first artificial neural network module 310, the second artificial neural network module 320, and the third artificial neural network module 330 in the automatic music transcription device according to FIG. However, as an embodiment, the artificial neural network module according to FIG. 7 is borrowed as the first artificial neural network module 310, and the second artificial neural network module 320 and the third artificial neural network module 330 according to FIG. An artificial neural network module may be employed. In this case, the first artificial neural network module 310 serves as an artificial neural network that generates learning data necessary for the second artificial neural network module 320 to learn, and the second artificial neural network module 320 substantially automatically transcribes vocals. At the same time as performing a role of performing a role of outputting the data to be learned by the third artificial neural network module 330 may be performed. That is, in this case, when the third artificial neural network module 330 performs learning, the second artificial neural network module 320 becomes a model serving as a teacher outputting more accurate information, and the third artificial neural network module 330 may model the role of a student in that learning is performed based on information output by the second artificial neural network module 320 . That is, in the case of the automatic music transcription device according to FIG. 12, the third artificial neural network module 330 performs learning based on a semi-supervised learning method.

Redundant descriptions of the first artificial neural network module 310, the second artificial neural network module 320, and the pre-processing module 340 are omitted, and the first post-processing module 351 and the second post-processing module 352 are also Since it has the same role as the post-processing module 350 described above, a detailed description thereof will be omitted.

The third artificial neural network module 330 takes third audio data in the frequency domain as fifth input information, and receives pitch information including pitch information of vocals in units of notes for the third audio data. A fifth artificial neural network output as fifth output information and intermediate output information output from a block constituting the fifth artificial neural network are set as fifth input information, and for the third audio data, in frame units, vocal It may include a sixth artificial neural network that outputs vocal information about the presence or absence of as sixth output information.

The third artificial neural network and the fourth artificial neural network of the second artificial neural network module 320 correspond to the first artificial neural network 311 and the second artificial neural network 312 of the first artificial neural network module 310, respectively. Correspondingly, most of the components are the same. However, as mentioned above, as one embodiment, the first artificial neural network module 310 borrows the artificial neural network module according to FIG. 7, and the second artificial neural network module 320 borrows the artificial neural network module according to FIG. 8. It can be.

Describing the learning method of the second artificial neural network module 320, the second artificial neural network module 320 uses the first learning data stored in the first memory module 210 as learning data and the second memory module 320. ) Learning may be performed based on at least one of the second learning data stored in ).

The fifth artificial neural network and the sixth artificial neural network of the third artificial neural network module 330 correspond to the artificial neural networks corresponding to the third artificial neural network and the fourth artificial neural network of the second artificial neural network module 320, respectively, and most of the components is the same However, as mentioned above, as one embodiment, the first artificial neural network module 310 borrows the artificial neural network module according to FIG. 7, and the second artificial neural network module 320 and the third artificial neural network module 330 An artificial neural network module according to FIG. 8 may be employed.

Describing the learning method of the third artificial neural network module 330, the second artificial neural network module 320 includes the third learning data stored in the third memory module 210 as learning data and the second artificial neural network module ( 320) is characterized in performing semi-supervised learning with the second artificial neural network module 320 as a teacher based on the output data.

Semi-supervised learning according to the present invention can be performed in several ways. First, the second artificial neural network module 320 and the third artificial neural network module 330 are trained on unlabeled data, respectively. There is a method of performing learning by comparing input and output results with each other and reducing the difference.

In the second method, the mixed data by adding random noisy data to unlabeled data is input to the second artificial neural network module 320 and the third artificial neural network module 330, which have been trained, respectively, and outputs the data. There is a method of performing learning by comparing the results to each other and reducing the difference.

In the third method, only unlabeled data is input to the second artificial neural network module 320, and random noisy data is added to the unlabeled data in the third artificial neural network module 330 to generate mixed data. There is a method of performing learning by comparing the output results of each input and reducing the difference.

All three methods described above obtained output results with relatively high accuracy compared to the prior art, and among the three methods, the third method in which random noise data was added only to the third artificial neural network module 330 corresponding to the student role was the most High accuracy results were obtained.

So far, the method and apparatus for automatic vocal transcription at the note level using an artificial neural network according to an embodiment have been studied in detail through the drawings.

A note-level automatic vocal transcription method and apparatus using an artificial neural network according to an embodiment uses the estimated vocal information in inferring pitch information including vocal information, and utilizes the estimated pitch information in inferring the vocal information. Therefore, there is an advantage of being able to infer pitch information and vocal information included in audio data more effectively than the prior art. Accordingly, there is no pre-processing step for separating vocals in transcription, so the amount of calculation is greatly reduced, and there is an advantage that vocal transcription can be performed with high accuracy even without separating vocals.

In addition, a method and apparatus for automatic vocal transcription of a note level using an artificial neural network according to an embodiment converts frame-level pitch information into pseudo-note level pitch information, and then performs note-level transcription of the converted data. Since it is used as training data of an artificial neural network, there is an advantage in that an artificial neural network capable of efficiently outputting note-level pitch information can be trained with less note-level training data.

In addition, since learning based on semi-supervised learning is performed in learning the artificial neural network, it is possible to efficiently learn the artificial neural network that outputs note-level pitch information with only a small amount of unlabeled data.

Accordingly, the note-level automatic vocal transcription method and apparatus using an artificial neural network according to an embodiment can perform flexible and accurate transcription for sound sources of various genres, and can directly generate vocal melodies from polyphonic music without a source separation preprocessing process. Since it is possible to transcribe, there is an advantage that the transcribing speed can be greatly improved compared to the prior art.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved. Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

100: user terminal 110: recording unit
120: communication unit 130: MIDI generation unit
140: display unit 200: music transcription device
300: processor 310: first artificial neural network module
320: second artificial neural network module 400: memory module
500: Additional service module 600: User management module

Claims (7)

one or more processors; and
A memory module for storing instructions executable by the one or more processors; includes,
the processor,
A first step that uses first audio data in the frequency domain as first input information and outputs pitch information including pitch information of vocals in units of frames for the first audio data as first output information. artificial neural network;
A pre-processing module for converting the first output information into first learning data including vocal information in units of notes;
a third artificial neural network that receives second audio data in the frequency domain as third input information and outputs pitch information including vocal pitch information in frame units for the second audio data as third output information; and
A post-processing module for transforming the third output information into note-based vocal information;
The third artificial neural network,
Characterized in that learning is performed for the third artificial neural network based on the first learning data,
A vocal transcription device that performs learning based on note-level audio data. According to claim 1,
The pre-processing module and the post-processing module,
Characterized in that pitch information in units of frames is converted into note-level pitch information using a pitch quantization method and a rhythm quantization method,
A vocal transcription device that performs learning based on note-level audio data. According to claim 1,
a first memory module that stores the first learning data; and
A second memory module including second learning data including pitch information labeled at the note level; further comprising,
A vocal transcription device that performs learning based on note-level audio data. According to claim 3,
The third artificial neural network,
Characterized in that learning is performed for the third artificial neural network based on the second learning data,
A vocal transcription device that performs learning based on note-level audio data. one or more processors; and
A memory module for storing instructions executable by the one or more processors; includes,
the processor,
A first step that uses first audio data in the frequency domain as first input information and outputs pitch information including pitch information of vocals in units of frames for the first audio data as first output information. artificial neural network;
A pre-processing module for converting the first output information into first learning data including vocal information in units of notes;
a third artificial neural network that receives second audio data in the frequency domain as third input information and outputs pitch information including vocal pitch information in frame units for the second audio data as third output information;
a fifth artificial neural network that takes third audio data in the frequency domain as fifth input information and outputs pitch information including vocal pitch information in frame units with respect to the third audio data as fifth output information; and
A first post-processing module for converting the third output information into second learning data including vocal information in units of notes;
The third artificial neural network,
Characterized in that learning is performed based on at least one of the second learning data and the third learning data,
A vocal transcription device that performs learning based on note-level audio data. According to claim 5,
A second post-processing module for transforming the fifth output information into note-based vocal information; further comprising,
A vocal transcription device that performs learning based on note-level audio data. In the learning method of a vocal transcription device using a processor containing an artificial neural network,
The processor takes first audio data in the frequency domain as first input information, and outputs pitch information including vocal pitch information in units of frames for the first audio data as first output information. A first output information outputting step of outputting the first output information using a first artificial neural network that:
a pre-processing step of converting, by the processor, the first output information into first training data including vocal information in units of notes;
a learning step in which the processor performs learning on a third artificial neural network based on the first learning data;
The third artificial neural network in which the processor receives second audio data in the frequency domain as third input information and outputs pitch information including vocal pitch information in frame units for the second audio data as third output information. a third output information output step of outputting the third output information by using; and
Characterized in that it comprises a; post-processing step of the processor converting the third output information into vocal information in units of notes,
A learning method of a vocal transcription device that performs learning based on note-level audio data.

Images