KR102555701B1 - Automatic synchronization between musical audio signals and lyrics using artificial intelligence - Google Patents

Abstract

An apparatus and method for automatic lyrics synchronization using artificial intelligence are provided. An automatic lyrics synchronizing device according to an embodiment is an automatic lyrics synchronizing device that automatically corrects lyric data related to a sound source and generates modified lyric data, comprising: a voice extraction unit that extracts voice from the sound source; a data pre-processing unit that performs pre-processing on lyric data to generate pre-processed lyric data; and a forced alignment performer generating the corrected lyrics data by forcibly aligning the voice and the preprocessed lyric data.

Description

Automatic synchronization between musical audio signals and lyrics using artificial intelligence

The present invention relates to a method and apparatus for automatically synchronizing lyrics using artificial intelligence, and relates to a method and apparatus for automatically synchronizing lyrics to a sound source using an artificial intelligence-based voice recognition apparatus and forced-alignment.

Along with the 4th industrial revolution, music sources in various languages and lyrics corresponding to the music sources are provided to users. In the case of overseas sound sources, lyrics are provided in the original foreign language, but when supplied to Korea, lyrics are provided in Korean notation so that the song can be sung along. Recently, with the craze of K-pop, there are many opposite cases, and the method of transliteration in the language as it is pronounced is called transliteration.

The full text of the lyrics may simply be provided together with the sound source, but the lyrics may be selectively output to the user by synchronizing the lyrics and the sound source through preliminary work. That is, lyrics corresponding to the reproduced sound source are highlighted, and a service is provided to the user in which the user identifies the corresponding lyrics and supports singing along.

However, synchronization of lyrics and sound sources is mostly performed manually, and a lot of manpower and time are required. Therefore, the lyrics synchronization service is performed mainly on popular sound sources, and not all sound sources are provided.

There is a demand for a device and method capable of automatically synchronizing lyrics corresponding to sound sources more efficiently and accurately so that lyrics synchronization services can be provided to various sound sources.

An object of the present invention is to provide an apparatus and method capable of automatically synchronizing lyrics corresponding to sound sources more efficiently and accurately.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

An automatic lyrics synchronizing apparatus according to some embodiments of the present invention is an automatic lyrics synchronizing apparatus for automatically modifying lyric data related to a sound source and generating modified lyric data, comprising: a voice extraction unit that extracts voice from the sound source; a data pre-processing unit that performs pre-processing on lyric data to generate pre-processed lyric data; and a forced alignment performer generating the corrected lyrics data by forcibly aligning the voice and the preprocessed lyric data.

In addition, the forced alignment performer performs the forced alignment based on the first language, and the data preprocessor converts the text composed of the second language of the lyrics data into the first language according to the pronunciation of the second language. 1 language conversion unit that performs preprocessing; and a token conversion unit configured to generate the pre-processed lyrics data by performing tokenization on the first pre-processed lyrics data.

Also, the token conversion unit may form a token to indicate the beginning of a word with a special symbol, and the language conversion unit may generate first preprocessing information performed on the lyric data.

Also, the forced alignment unit may generate the corrected lyric data by inversely converting the first pre-processed text into the second language using the first pre-processing information.

In addition, the forced alignment performing unit may include a voice feature extraction unit extracting voice features of the voice recognition text; a speech section detection unit configured to detect a speech section of the speech recognition text and generate speech section information; a forced alignment unit generating voice alignment information by forcibly aligning the extracted voice features and the preprocessed lyric data; and a correction information generating unit configured to generate corrected alignment information by comparing the voice alignment information with the voice interval information.

Also, the forced alignment unit may be configured to generate the voice alignment information again by receiving feedback of the corrected alignment information.

In addition, the forced alignment unit analyzes the extracted voice features and the preprocessed lyric data to extract voice alignment information in units of character elements, and voice alignment information corresponding to a higher unit based on the extracted voice alignment information in units of character elements. can create

In addition, the sound source is provided by combining the voice and musical instrument sound in a stereo format, and the voice extraction unit extracts the voice from the sound source through a deep learning-based voice separation model, and the voice separation model is used to extract the musical instrument from the sound source. It may be in a learned state to extract only the voice by filtering the sound.

An automatic lyrics synchronization method according to some embodiments of the present invention is an automatic lyrics synchronization method performed by an automatic lyrics synchronization device, comprising the steps of inputting a sound source and lyrics data related to the sound source; extracting voice from the sound source; performing pre-processing on the lyric data to generate pre-processed lyric data; and generating the modified lyrics data by forcibly aligning the voice and the preprocessed lyrics data.

A computer program according to some embodiments of the present invention is combined with hardware and stored in a medium to execute the automatic lyrics synchronization method.

The apparatus and method for synchronizing automatic lyrics according to an embodiment of the present invention can output lyrics in units of words and tokens corresponding to lower units of a sentence by sorting several higher units starting from a grapheme unit. Therefore, it can be supported to quickly generate lyrics in a form desired by the creator automatically by holding the starting and ending points of the approximate position, transcribing the voice into sentences of units in which the context is connected, and displaying only the parts to be separated between words. .

The apparatus and method for synchronizing automatic lyrics according to an embodiment of the present invention may output lyrics in units of words, tokens, and lines corresponding to lower units of a sentence by performing sorting on several upper units from the result of forcibly sorting on a grapheme basis. there will be Therefore, it is supported to quickly generate lyrics in the form of synchronizing information (timestamp) desired by the generator automatically by displaying the part where lyrics data should be separated into sentences of units connected with context with symbols that act as line changes. It can be.

In addition, it is possible to minimize the number of out-of-vocabulary (OOV) words through tokenization of the text of the lyric data, so that forced sorting can be performed more efficiently.

In addition, the apparatus and method for automatic lyrics synchronization according to an embodiment of the present invention can support more efficient forced alignment by automatically performing language conversion on lyrics data corresponding to a reference language for which forced alignment is performed. In addition, while performing tokenization on the text of the lyric data, it is possible to support more efficient execution of forced alignment by displaying the start position of a word.

In addition, the automatic lyrics synchronizing apparatus and method according to an embodiment of the present invention is configured to compensate for the error part of forced-alignment caused by various voice variations of a song (voice) through voice section information, so that more accurate forced-alignment It can support sorting being performed.

1 is an exemplary diagram for explaining the concept of an automatic lyrics synchronization device according to some embodiments of the present invention.
2 is an exemplary view schematically showing the configuration of an automatic lyrics synchronization device according to some embodiments of the present invention.
3 is an exemplary diagram for explaining the configuration and operation of a voice extraction unit according to an embodiment.
4 is a block diagram showing the configuration of a data pre-processing unit according to an embodiment.
5 is an exemplary diagram for explaining the configuration and operation of a forced alignment performer according to an embodiment.
6 is an exemplary view for explaining the operation of the forced alignment unit.
7 illustratively shows results of forced sorting in grapheme units.
8 illustratively shows results of forced sorting in word units.
9 illustratively shows voice alignment information in sentence units.
10 shows voice section detection information by way of example.
FIG. 11 exemplarily shows voice alignment information obtained by correcting the voice alignment information of FIG. 9 based on the voice section detection information of FIG. 10 .
12 is a flowchart of an automatic lyrics synchronization method according to another embodiment of the present invention.

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. According to the principle that an inventor may define a term or a concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical spirit of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all of the technical spirit of the present invention, so they can be replaced at the time of the present application. It should be understood that there may be many equivalents and variations and applicable examples.

Terms such as first, second, A, and B used in this specification and claims may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

Terms used in this specification and claims are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "include" or "having" in this application do not exclude in advance the possibility of existence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, with reference to FIGS. 1 to 11 , an automatic lyrics synchronization method and apparatus according to some embodiments of the present invention will be described.

1 is an exemplary diagram for explaining the concept of an automatic lyrics synchronization device according to some embodiments of the present invention. 2 is an exemplary view schematically showing the configuration of an automatic lyrics synchronization device according to some embodiments of the present invention. 3 is an exemplary diagram for explaining the configuration and operation of a voice extraction unit according to an embodiment. 4 is a block diagram showing the configuration of a data pre-processing unit according to an embodiment. 5 is an exemplary diagram for explaining the configuration and operation of a forced alignment performer according to an embodiment. 6 is an exemplary view for explaining the operation of the forced alignment unit. 7 illustratively shows results of forced sorting in grapheme units. 8 illustratively shows results of forced sorting in word units. 9 illustratively shows voice alignment information in sentence units. 10 shows voice section detection information by way of example. FIG. 11 exemplarily shows voice alignment information obtained by correcting the voice alignment information of FIG. 9 based on the voice section detection information of FIG. 10 .

Referring to FIG. 1 , the automatic lyrics synchronization device 10 may receive a sound source and lyrics data.

The sound source may be music having a reproduction length of a predetermined time. The sound source may include a song corresponding to the singer's voice and an instrumental sound corresponding to the song. A singer's song may consist of at least one language. For example, a song may include voices in both Korean and English, but is not limited thereto. Lyrics data may represent information related to voice among sound sources. Lyrics data may correspond to visual information for visually conveying voice, which is auditory information. In an embodiment, the lyric data may be configured to convey visual information corresponding to voice to a user through at least one language.

In an embodiment, the sound source may have a compressed sound source format provided as a streaming service. For example, the sound source may have one of audio codecs such as ac3, mp3, aac, wma, flac, opus, and wav. Lyrics data may include a plurality of lines in units of phrases and sentences. In addition, the lyric data may include time stamps corresponding to start and end times for each line, and may be configured in one of LRC, SMI, and SRT formats. Here, the lyric data and sound source (specifically, voice) may be out of sync. In an embodiment, the plurality of lines of lyric data may be in a state in which time stamps are not set, but is not limited thereto. In some embodiments, time stamps are set for a plurality of lines of lyric data, but synchronization between the set time stamp and the sound source may not be performed, and thus may be out of sync.

The automatic lyrics synchronization device 10 according to the embodiment receives lyric data that is not synchronized with the sound source, compares and analyzes the sound source and lyric data, automatically synchronizes the lyric data with the sound source, generates corrected lyrics data, and generates modified lyrics data. Lyrics data can be output. The automatic lyrics synchronization device 10 may generate modified lyrics data by newly setting time stamps corresponding to sound sources in a plurality of lines of input lyrics data or by modifying time stamps to correspond to sound sources.

Referring to FIG. 2 , the automatic lyrics synchronization device 10 includes a voice extraction unit 100 , a data pre-processing unit 110 and a forced alignment unit 120 . The automatic lyrics synchronization device 10 according to the present embodiment may be entirely hardware, or may have an aspect of being partially hardware and partially software. For example, an automatic lyrics synchronization device and each module included therein may collectively refer to a device for exchanging data of a specific format and content in an electronic communication method and software related thereto. In this specification, terms such as “unit”, “module”, “server”, “device”, “device” or “terminal” refer to a combination of hardware and software driven by the hardware. It is intended to refer to For example, the hardware may be a data processing device including a CPU, GPU, NPU, or other processor. Also, software driven by hardware may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

In addition, each module constituting the automatic lyrics synchronization device 10 is not necessarily intended to refer to a separate physically distinct component. The voice extraction unit 100, the data pre-processing unit 110, and the forced alignment execution unit 120 are shown as separate blocks, but this only functionally divides the configuration of the automatic lyrics synchronization device by the operation to be executed. Therefore, depending on the embodiment, some or all of the voice extractor 100, the data preprocessor 110, and the forced alignment performer 120 may be integrated in the same device, and one or more may be physically integrated with other units. It may be implemented as a separate device distinguished by , or may be components communicatively connected to each other under a distributed computing environment.

The voice extraction unit 100 extracts voice from a sound source. The voice extraction unit 100 is configured to extract only voice from an input sound source. It may be effective to apply the forced-alignment of lyric data performed by the forced-aligner 120 to be described later only to the voice. Therefore, forced alignment can be effectively performed only when noise other than voice and musical instrument sounds are effectively filtered out of the sound source, so that voice and lyrics can be synchronized accurately.

In an embodiment, the input sound source may be in a state in which voice and musical instrument sound are combined in a stereo format. That is, since the input sound source is not provided as a multi-track sound source composed of different tracks for voice and instrument sound, it may be difficult to extract the sound simply by separating the tracks.

In an embodiment, the voice extractor 100 may include a deep learning-based voice separation model, and may extract only voice from an input sound source through the voice separation model. The speech separation model may be constructed based on a deep learning algorithm composed of a hierarchical structure including an input layer, an output layer, and at least one intermediate layer (or hidden layer) between the input layer and the output layer. The deep learning algorithm, based on such a multi-layered structure, can derive highly reliable results through learning that optimizes weights of activation functions between layers.

Deep learning algorithms applicable to the present invention include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), and a long short term memory (LSTM). ), Bidirectional Short Term Memory (BLSTM), etc. Here, since speech is time-series data, a speech separation model based on LSTM or BLSTM specialized for repetitive and sequential data learning It may be implemented, but is not limited thereto.

The voice extraction unit 100 can effectively filter instrumental sounds from sound sources through the above-described deep learning algorithm, and extracts only voices.

FIG. 3A illustrates an input sound source as an example, and FIG. 3B illustrates a state in which musical instrument sounds are filtered and only voice is separated from the input sound source through the above-described voice separation model. 3A and 3B illustratively show the waveform of a sound source over time. Referring to FIG. 3B, compared to FIG. 3A, only voices separated with musical sounds other than voices completely removed are illustratively shown as waveforms. It can be seen that it is illustrated.

In an embodiment, the voice extractor 100 may further perform a preprocessing process of changing the format of the separated voice into a format suitable for data processing described below. In an embodiment, the pre-processing process may include changing a sampling rate (sampling frequency) region of voice or changing stereo to mono so that it can be used in the forced alignment unit 120, but is not limited thereto. In the voice extraction unit 100, the extracted voice may be provided to the forced alignment performing unit 120.

The data pre-processing unit 110 may perform data pre-processing for force-alignment.

Referring to FIG. 4 , the data pre-processing unit 110 includes a language conversion unit 112 and a token conversion unit 114 .

In an embodiment, the audio of the sound source may be composed of at least two languages, and the lyric data may also be composed of two languages. The forced alignment performed by the forced alignment performer 120 may be performed based on one language without using a pronunciation dictionary. For example, the voice may be content in which a first language is mainly used, but a second language is partially used, and the lyric data includes both the first language and the second language in correspondence with the language configuration of the voice. can be Illustratively, the first language may be Korean and the second language may be English, but is not limited thereto. Since the forced alignment performer 120 is configured to perform forced alignment between voice and text based on the first language, a preprocessing process of converting the lyric data into the first language may be required.

The language conversion unit 112 may perform an operation of converting the above-described lyric data into a first language. Specifically, the language conversion unit 112 may perform preprocessing of converting a sentence composed of a second language into a first language according to a pronunciation of the second language. For example, "I believe in you" can be converted to "I believe in you." The language conversion unit 112 may generate first pre-processed lyrics data by performing a language conversion task of converting lyrics data into one language. That is, the first pre-processed lyric data may indicate a state in which a first pre-process for converting into one language has been performed. The language conversion unit 112 may perform first pre-processing for each sentence of lyric data. The language conversion unit 112 may generate first pre-processing information, which is information about text on which the first pre-processing has been performed, in lyrics data. Based on the first pre-processing information, inverse conversion may be performed on the first pre-processed text when the modified lyric data is generated through forced alignment later.

In an embodiment, the forced alignment performer 120 may perform forced-alignment based on a token unit or grapheme unit corresponding to a lower unit than a word unit. In particular, when the language to be aligned is Korean, the number of words to be recognized is too large due to a large number of postpositions and semantic changes, so a pre-processing task of separating recognition target units into token units may be required.

The token conversion unit 114 may perform a process of tokenizing the first pre-processed lyric data. In an embodiment, the token conversion unit 114 may perform tokenization on scripts included in each pre-processed lyric data. As it is divided into units smaller than word units through tokenization, it is possible to reduce the vocabulary subject to forced alignment and minimize the number of out-of-vocabulary (OOV) words. Forced sorting can be performed more effectively.

In some embodiments, token conversion unit 114 may be configured to include a custom dictionary for performing tokenization. The dictionary is a dictionary for performing tokenization and may correspond to a dictionary built while preserving syllables or grapheme included in the script without loss.

Additionally, in some embodiments, token conversion unit 114 may form tokens using tokens to indicate the beginning of a word and special symbols to indicate a token that is not the beginning of a word. For example, "_for the first time, yes, in a few days it will be fine" is tokenized as "_for the first time _yes _for _days _after _it will be fine", but the beginning of a word in the original sentence is marked with a special symbol such as _. Tokenization can be performed. In this way, since the start position of the divided text is identified as a special symbol, detokenization can be performed on a word-by-word basis on the result of the token unit extracted as a result of forced alignment without ambiguity.

The token conversion unit 114 may generate second pre-processed lyrics data by tokenizing the first pre-processed lyrics data as described above. The generated second pre-processed lyric data may be provided to the forced alignment performer 120 .

The forced alignment performer 120 may generate modified lyrics data by performing forced alignment based on the voice and lyrics data. The forced alignment performer 120 may extract voice alignment information including information of character elements constituting the lyrics data and information about which character elements a specific frame of voice corresponds to, and based on this, voice and lyrics data may be extracted. Corrected lyric data may be generated by synchronization.

Referring to FIG. 5 , the forced alignment performing unit 120 includes a voice feature extraction unit 122, a voice section detection unit 124, a forced alignment unit 126, and a correction information generating unit 128.

The speech feature extraction unit 122 may extract a feature vector for speech. In an embodiment, the feature vector may be MFCC (Mel Frequency Cepstral Coefficients) or a filter bank, but is not limited thereto. The voice feature extraction unit 122 may divide the voice into voice frame units based on the time window and extract voice features based on the voice frame unit. Specifically, the MFCC may divide the “voice” into “voice” frame units based on a time window, and convert the “voice” frame into a frequency domain using FFT (Fast Fourier Transform).

In addition, by using a filter bank, MFCC divides the voice frame into signals of several frequency bands, calculates the energy of the signal of each frequency band, and applies DCT (Discrete Cosine Transform) to the calculated energy. can be obtained.

Also, in some embodiments, the voice feature may be a frequency feature of the voice obtained by obtaining a spectrogram from the voice and extracted from the spectrogram. In the spectrogram generated, the larger the amplitude value, the larger the bin size, and the smaller the amplitude value, the smaller the bin size. The voice feature extraction unit 122 may extract the frequency feature information indicated in the spectrogram as a voice feature.

Referring to FIG. 6 , the forced alignment unit 126 may forcibly align text of lyric data corresponding to extracted voice features. In an embodiment, the forced sorting unit 126 may perform forced sorting based on the first language, and perform first preprocessing and tokenization of lyrics data input to the forced sorting unit 126 to be converted into the first language. It may be lyric data on which the second pre-processing has been performed. That is, more accurate and efficient forced alignment can be performed in a state in which units to be recognized are separated into token units in advance, and the start position of a word is confirmed with a special symbol.

The compulsory alignment unit 126 may generate voice alignment information by analyzing voice characteristics and the second preprocessed lyrics data subjected to language conversion and tokenization.

7 shows phonetic alignment information in grapheme units for the lyric sentence "I'll be your umbrella on a rainy day and a light when it's dark." In the voice alignment information, the X axis may mean time (s), and the Y axis may mean normalized voice amplitude with a maximum value of 1 and a minimum value of -1. As shown in FIG. 7 , the forced alignment performer 120 may extract voice alignment information in units of character elements of lyrics data. In addition, the forced alignment performer 120 may extract a result by token, a result by word, and a result by sentence corresponding to a higher unit based on the phonetic alignment information extracted in units of graphemes. FIG. 8 shows voice alignment information in units of words for the sentence of FIG. 7 . As sorting is performed for several upper units starting from the grapheme unit, it is possible to output lyrics in units of words and tokens corresponding to lower units of sentences instead of outputting lyrics in units of sentences.

Here, the song (voice) included in the sound source may further include a characteristic element different from that of a general person's speech (voice). Singing (voice) may cause distortion of voice due to various vocalizations such as vibration, pronunciation variation, falsetto, chorus, and voice modulation. Forced-alignment is to align the text corresponding to what the voice says, and it can achieve a certain level of performance (accuracy), but there is difficulty in perfectly recognizing it.

The forced alignment performing unit 120 according to the present embodiment detects a voice section and a non-speech section of the extracted voice, and further performs a process of correcting voice alignment information based on the detected result, so as to perform the above-described song ( voice) may be configured to output voice alignment information reflecting characteristics. That is, forced alignment for songs can be supplemented through speech activity detection technology.

Again referring to FIG. 5 , the speech section detection unit 124 may generate speech section information by analyzing the extracted speech. The speech section detection unit 124 may include a pre-learned non-speech section detection model. The non-speech section detection model may be a deep learning-based learning model, but is not limited thereto. In an embodiment, the non-voice section detection model may be in a pre-learned state to detect non-voice sections, which are noises caused by various vocalizations such as vibration, pronunciation variation, virtual, chorus, voice modulation, and the like. A section excluding the non-voice section identified through the non-voice section detection model may be a speech section. The speech segment detection unit 124 may generate speech segment information identifying the speech segment and the non-speech segment, and the generated speech segment information may be provided to the correction information generation unit 128.

The correction information generating unit 128 may receive the voice section information and the voice alignment information, and generate the correction alignment information by comparing the voice alignment information and the voice interval information. The correction information generation unit 128 utilizes the non-speech section included in the speech section information to generate correction information that compensates for the error part of forced-alignment caused by various voice variations of the song (speech). can Here, the correction information generation unit 128 may generate correction alignment information excluding a section corresponding to the non-speech section from the speech alignment information.

9 is an example image showing voice alignment information, and shows voice alignment information for the sentence “Hello, it's not easy, I'm busy.” FIG. 10 shows a speech section and a non-speech section for the sentence corresponding to FIG. 9 . In FIG. 10 , the portion identified as a red image may be a voice section, and the remaining sections may be a non-voice section. Illustratively, the non-voice period may be a period in which vibration is performed. The correction information generation unit 128 may generate correction alignment information for processing so that a section corresponding to a non-speech section in the voice alignment information is excluded from subsequent voice alignment. The correction information generation unit 128 may feed back the corrected alignment information to the forced alignment unit 126, and the forced alignment unit 126 refers to the corrected alignment information to perform forced alignment on voice features and lyric data again. and generate voice alignment information. FIG. 11 is a result of performing forced alignment on the line “Hello, it's not easy, I'm busy” again with reference to corrected alignment information generated based on the non-speech section of FIG. 10 . Comparing FIG. 11 with FIG. 9, as the non-speech section is excluded by the modified alignment information, the sentence "It's not easy, I'm busy", which was overlapped and sorted in the non-speech section, has a timestamp so that it starts after the non-speech section ends. will be moved

The forced alignment performer 120 may generate corrected lyrics data by correcting the time stamp of the lyrics data based on the regenerated voice alignment information. That is, modified lyric data aligned to correspond to voice data in units of character elements may be generated. The forced alignment performing unit 120 may perform inverse transformation on the first pre-processed text based on the first pre-processing information, and the modified lyric data is configured to include both the first language and the second language as in the prior art. .

The apparatus for synchronizing automatic lyrics according to an embodiment of the present invention can output lyrics in units of words and tokens corresponding to lower units of a sentence by sorting several upper units starting from a grapheme unit. Therefore, it can be supported to quickly generate lyrics in a form desired by the creator automatically by holding the starting and ending points of the approximate position, transcribing the voice into sentences of units in which the context is connected, and displaying only the parts to be separated between words. .

In addition, more efficient forced alignment may be supported by automatically performing language conversion on lyrics data corresponding to a reference language in which forced alignment is performed.

In addition, while performing tokenization on the text of the lyric data, it is possible to support more efficient execution of forced alignment by displaying the start position of a word.

In addition, it is configured to compensate for errors in forced-alignment caused by various voice variations of songs (voices), so that more accurate forced-alignment can be performed.

12 is a flowchart of an automatic lyrics synchronization method according to another embodiment of the present invention. The automatic lyrics synchronization method according to FIG. 12 is performed by the automatic lyrics synchronization device 10 of FIGS. 1 to 11 described above. 1 to 11 and related descriptions may be referred to for description and understanding of this embodiment.

Referring to FIG. 12 , the automatic lyrics synchronization method according to the embodiment includes inputting a sound source and lyric data related to the sound source (S110); Extracting voice from the sound source (S120); generating preprocessed lyrics data by performing preprocessing on the lyrics data (S130); and generating the corrected lyrics data by forcibly aligning the voice and the preprocessed lyrics data (S140).

In the step S140, the forced sorting may be performed based on the first language, and in the step S130, the text composed of the second language of the lyrics data is converted to the first language according to the pronunciation of the second language. performing a first pre-processing of conversion; and generating the pre-processed lyrics data by tokenizing the first pre-processed lyric data.

The generating of the preprocessed lyric data by performing the tokenization may include forming a token to indicate a start of a word with a special symbol.

The generating of the first pre-processed lyric data includes generating first pre-processed information performed on the lyric data, and the step (S140) includes the first pre-processed text using the first pre-processed information. and generating the modified lyric data by inversely converting the lyric into the second language.

The step (S140) includes extracting voice features from the extracted voice; generating voice section information by detecting a voice section from the extracted voice; generating voice alignment information by forcibly aligning the extracted voice features and the preprocessed lyric data; and generating corrected alignment information by comparing the voice alignment information with the voice section information.

The step S140 may further include regenerating the voice alignment information by receiving feedback of the corrected alignment information.

In an embodiment, the generating voice alignment information may include extracting voice alignment information in units of character elements by analyzing the extracted voice characteristics and the preprocessed lyric data, and extracting voice alignment information in units of character elements in a higher unit based on the voice alignment information extracted in units of character elements. It may include generating corresponding voice alignment information.

In the embodiment, the sound source is provided by combining the voice and musical instrument sound in a stereo format, and the step (S110) includes extracting the voice from the sound source through a deep learning-based voice separation model, and the voice separation The model may be in a state in which it has been learned to extract only the voice by filtering the musical instrument sound from the sound source.

The automatic lyrics synchronization method according to the embodiment may be implemented in the form of a computer-readable medium storing instructions and data executable by a computer. In this case, instructions and data may be stored in the form of program codes, and when executed by a processor, a predetermined program module may be generated to perform a predetermined operation. Also, computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, a computer-readable medium may be a computer recording medium, which is a volatile and non-volatile memory implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It can include both volatile, removable and non-removable media. For example, the computer recording medium may be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD, and Blu-ray disc, or a memory included in a server accessible through a network.

Also, the automatic lyrics synchronization method according to the embodiment may be implemented as a computer program (or computer program product) including instructions executable by a computer. A computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, object-oriented programming language, assembly language, or machine language. . Also, the computer program may be recorded on a tangible computer-readable recording medium (eg, a memory, a hard disk, a magnetic/optical medium, or a solid-state drive (SSD)).

Accordingly, the automatic lyrics synchronization method according to the embodiment may be implemented by executing the computer program as described above by a computing device. A computing device may include at least some of a processor, a memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components are connected to each other using various buses and may be mounted on a common motherboard or mounted in any other suitable manner.

Here, the processor may process commands within the computing device, for example, to display graphic information for providing a GUI (Graphic User Interface) on an external input/output device, such as a display connected to a high-speed interface. Examples include instructions stored in memory or storage devices. As another example, multiple processors and/or multiple buses may be used along with multiple memories and memory types as appropriate. Also, the processor may be implemented as a chipset comprising chips including a plurality of independent analog and/or digital processors.

Memory also stores information within the computing device. In one example, the memory may consist of a volatile memory unit or a collection thereof. As another example, the memory may be composed of a non-volatile memory unit or a collection thereof. Memory may also be another form of computer readable medium, such as, for example, a magnetic or optical disk.

Also, the storage device may provide a large amount of storage space to the computing device. A storage device may be a computer-readable medium or a component that includes such a medium, and may include, for example, devices in a storage area network (SAN) or other components, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory, or other semiconductor memory device or device array of the like.

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

Claims (10)

An automatic lyrics synchronizing device for generating modified lyrics data by automatically modifying lyrics data related to a sound source, the automatic lyrics synchronizing device comprising:
a voice extraction unit extracting voice from the sound source;
A data pre-processor that performs pre-processing on lyric data to generate pre-processed lyric data, wherein lyric data including text in a first language and text in a second language different from the first language is composed of the second language a language conversion unit which performs a first pre-processing of converting text into the first language according to the pronunciation of the second language; and a token conversion unit configured to perform tokenization on the first pre-processed lyric data to generate the pre-processed lyric data; and
A forced alignment performer generating the corrected lyrics data by forcibly aligning the voice and the preprocessed lyric data based on the first language,
Automatic Lyrics Synchronizer.
delete According to claim 1,
The token conversion unit forms a token to indicate the start of a word with a special symbol;
wherein the language conversion unit generates first preprocessing information performed on the lyrics data;
Automatic Lyrics Synchronizer.
According to claim 3,
The forced alignment performer inversely converts the first pre-processed text into the second language using the first pre-processing information to generate the corrected lyric data;
The modified lyric data includes text composed of the first language and text composed of the second language.
Automatic Lyrics Synchronizer.
According to claim 1,
The forced sorting unit,
a voice feature extraction unit extracting voice features from the extracted voice;
a voice section detection unit configured to detect a voice section of the extracted voice and generate voice section information;
a forced alignment unit generating voice alignment information by forcibly aligning the extracted voice features and the preprocessed lyric data; and
a correction information generation unit configured to compare the voice alignment information with the voice interval information to generate correct alignment information;
Automatic Lyrics Synchronizer.
According to claim 5,
The forced alignment unit,
And configured to generate the voice alignment information again by receiving feedback of the corrected alignment information.
Automatic Lyrics Synchronizer.
According to claim 5,
The forced alignment unit
Analyzing the extracted voice characteristics and the preprocessed lyrics data to extract voice alignment information in units of character elements, and generating voice alignment information corresponding to a higher unit based on the voice alignment information extracted in units of character elements;
Automatic Lyrics Synchronizer.
According to claim 1,
The sound source is provided by combining the voice and instrument sound in a stereo format,
The voice extraction unit,
extracting the voice from the sound source through a deep learning-based voice separation model;
The voice separation model is in a state in which it is learned to extract only the voice by filtering the musical instrument sound from the sound source.
Automatic Lyrics Synchronizer.
A lyrics automatic synchronization method performed in an automatic lyrics synchronization device, the method comprising:
inputting a sound source and lyric data related to the sound source;
extracting voice from the sound source;
Generating pre-processed lyric data by performing pre-processing on the lyric data, wherein the lyric data including text in a first language and text in a second language different from the first language includes text in a second language. performing a first pre-processing of converting into the first language according to the pronunciation of the second language; and performing tokenization on the first pre-processed lyric data to generate the pre-processed lyric data; and
Generating modified lyrics data by forcibly aligning the voice and the preprocessed lyrics data based on the first language,
How to automatically synchronize lyrics.
A computer program stored in a medium for executing the automatic lyrics synchronization method according to claim 9 in combination with hardware.

Images