JP5098932B2 - Lyric data display device, lyrics data display method, and lyrics data display program - Google Patents

Description

  The present invention relates to a lyric data display device, a lyric data display method, and a lyric data display program for reproducing a recording medium on which music content is recorded and displaying lyrics at an appropriate timing in accordance with the reproduction.

In recent years, with the spread of services via the Internet, it has become possible to acquire the lyrics of music data using a search engine or the like. However, most of the lyrics that can be acquired are not provided with the timing (synchronization) information of the lyrics display corresponding to the reproduction time of the music data. Therefore, in order to display appropriate lyrics in accordance with the song while reproducing the music data, it is necessary to create timing data in advance. When inputting this timing data manually, a very complicated operation must be performed. In order to solve this problem, a method of automatically creating timing data has been proposed. For example, in the method of Patent Document 1, first, vocal sound is extracted from an acoustic signal by a bandpass filter, and the extracted vocal sound is encoded by speech recognition. Next, based on the coded vocal sound, the phrase of matching lyrics is searched and the found phrase is displayed.
JP 2002-169578 A

  However, in the method according to Patent Document 1, for example, if musical sounds of musical instruments in the vocal band such as piano and guitar are present in the musical composition data, it is difficult to extract only vocal voices. If it is not limited music data such as data or a cappella, there is a problem that the extraction accuracy is lowered and vocal voice cannot be extracted.

  In particular, since vocal speed, accent speed, frequency band, and the like are different from those of normal conversation, it is difficult for vocal voice to be processed correctly by a voice recognition engine.

  Accordingly, the present invention provides a lyric data display device and lyric data that can appropriately display lyrics in accordance with the reproduction time of the music data, even if the music data is not provided with timing data for displaying the lyrics. A display method and a lyric data display program are provided.

In order to solve the above problems, the present invention provides the following apparatus, method, and program.
(1) A singing time data storage unit that stores singing time data that is a start time and an ending time of the singing section for each singing section that is an existing section of singing data in the music data;
A lyric data storage unit for storing lyric data of the music data described as text data for each singing section;
A phoneme dictionary storage unit that stores a phoneme dictionary for converting characters or words into phoneme data, which is a minimum speech unit that works to distinguish the meaning of the character or word;
From the phoneme data, a mora number calculation rule storage unit that stores a mora number calculation rule for calculating the number of mora that is a segmental unit of a sound having a certain temporal length in pronunciation;
Reading the lyrics data from the lyrics data storage unit, referring to the phoneme dictionary storage unit, and converting the lyrics data into the phoneme data;
Read the phoneme data created by the phoneme data conversion unit with reference to the mora number calculation rule, and the number of mora in the singing section and the number of mora in each row of the singing section A mora number calculation unit for calculating the number of row mora;
The singing time data is read from the singing time data storage unit, the time length of the singing section is calculated, and the section mora number calculated by the mora number calculating unit is obtained by dividing by the time length. A mora speed calculator for calculating a mora speed for each singing section;
A display timing calculation unit for calculating a display timing for each line of the lyrics data based on the mora speed calculated by the mora speed calculation unit and the line mora number calculated by the mora number calculation unit;
Based on the display timing calculated by the display timing calculation unit, a display control unit for displaying the lyrics data in synchronization with the reproduction of the music data;
A lyric data display device.
(2) A singing time data storage unit that stores singing time data that is a start time and an ending time of the singing section for each singing section that is an existing section of singing data in the music data;
A lyrics data storage unit for storing lyrics data of the music data described as text data;
A phoneme dictionary storage unit that stores a phoneme dictionary for converting characters or words into phoneme data, which is a minimum speech unit that works to distinguish the meaning of the character or word;
From the phoneme data, a mora number calculation rule storage unit that stores a mora number calculation rule for calculating the number of mora that is a segmental unit of a sound having a certain temporal length in pronunciation;
Reading the lyrics data from the lyrics data storage unit, referring to the phoneme dictionary storage unit, and converting the lyrics data into the phoneme data conversion unit;
Read the phoneme data created by the phoneme data conversion unit with reference to the mora number calculation rule, and the total number of mora that is the number of mora of the entire lyrics data and the number of mora for each row of the lyrics data A mora number calculating unit for calculating the number of row mora,
The singing time data is read from the singing time data storage unit, the time length of each singing section and the total time length that is the total value thereof are calculated, and the total number of mora calculated by the mora number calculating unit is The average mora speed obtained by dividing by the total time length is calculated, and the estimated mora number of each singing section obtained by dividing the time length of each singing section by the average mora speed and the number of row mora Based on the singing section dividing unit for making the lyrics data correspond to each singing section for each line,
The number of mora for each singing section is calculated based on the correspondence information between the lyric data obtained by the singing section dividing unit and the singing section and the number of row mora calculated by the mora number calculating unit. And a mora speed calculation unit for calculating a mora speed for each singing section, which is obtained by dividing the time length of the singing section calculated by the singing section dividing unit,
A display timing calculation unit for calculating a display timing for each line of the lyrics data based on the mora speed calculated by the mora speed calculation unit and the line mora number calculated by the mora number calculation unit;
Based on the display timing calculated by the display timing calculation unit, a display control unit for displaying the lyrics data in synchronization with the reproduction of the music data;
A lyric data display device.
(3) In the lyrics data display device according to (1) or (2 ) above ,
further,
A kanji dictionary storage unit for storing one or more hiragana corresponding to kanji and how to read the kanji,
A hiragana conversion unit that converts kanji in the lyrics data acquired from the lyrics data storage unit into hiragana with reference to the kanji dictionary storage unit,
The phoneme data converter is
A lyric data display device that refers to the phoneme dictionary storage unit, converts the lyric data converted into hiragana by the hiragana conversion unit into the phoneme data, and creates a phoneme string.
(4) Read the lyrics data of the music data described as text data composed of one or more lines corresponding to the singing section, which is the existing section of the singing data in the music data, and the character or word of the word Creating a phoneme time-series data obtained by converting the lyric data into the phoneme data with reference to a phoneme dictionary for converting into phoneme data that is a minimum speech unit that functions to distinguish meaning;
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the number of section mora that is the number of mora in the singing section and each row of the singing section Calculating the number of row mora which is the number of mora of
Calculating the mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
Calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
A method for displaying lyric data.
(5) Read the lyric data of music data described as text data composed of one or more lines, and convert the characters or words into phoneme data which is the smallest speech unit that works to distinguish the meaning of words Creating phoneme time-series data by referring to a phoneme dictionary for conversion and converting the lyrics data into the phoneme data;
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the total number of mora that is the number of mora of the entire music data and the lyrics data Calculating the number of row mora, which is the number of mora in each row;
A step of calculating an average mora speed obtained by dividing the total number of mora by a total value of time lengths of each singing section which is an existing section of singing data in the music data;
Calculating an estimated number of mora for each song section obtained by dividing the time length of each song section by the average mora speed;
Creating correspondence information between each line of the lyrics data and each song section based on the estimated number of mora and the number of line mora;
Based on the correspondence information and the number of row mora, calculating a section mora number that is the number of mora for each singing section;
A mora speed calculating step for calculating a mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
A timing calculating step for calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
A method for displaying lyric data.
(6) Read the lyrics data of the music data described as text data composed of one or more lines corresponding to the singing section that is the existing section of the singing data in the music data, and the character or word of the word Creating a phoneme time-series data obtained by converting the lyric data into the phoneme data with reference to a phoneme dictionary for converting into phoneme data that is a minimum speech unit that functions to distinguish meaning;
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the number of section mora that is the number of mora in the singing section and each row of the singing section Calculating the number of row mora which is the number of mora of
Calculating the mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
Calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
Lyrics data display program that makes a computer execute.
(7) Read the lyrics data of music data described as text data composed of one or more lines, and convert the characters or words into phoneme data that is the smallest speech unit that has the function of distinguishing the meaning of words. Creating phoneme time-series data by referring to a phoneme dictionary for conversion and converting the lyrics data into the phoneme data;
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the total number of mora that is the number of mora of the entire music data and the lyrics data Calculating the number of row mora, which is the number of mora in each row;
A step of calculating an average mora speed obtained by dividing the total number of mora by a total value of time lengths of each singing section which is an existing section of singing data in the music data;
Calculating an estimated number of mora for each song section obtained by dividing the time length of each song section by the average mora speed;
Creating correspondence information between each line of the lyrics data and each song section based on the estimated number of mora and the number of line mora;
Based on the correspondence information and the number of row mora, calculating a section mora number that is the number of mora for each singing section;
A mora speed calculating step for calculating a mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
A timing calculating step for calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
Lyrics data display program that makes a computer execute.

  According to the lyrics data display device of the present invention, it is possible to give the timing information of the display start of the lyrics line to the lyrics data without the synchronization information of the music data and the lyrics, without the user performing a special work, An appropriate lyrics line can be displayed in accordance with the music data to be reproduced. In addition, it is possible to easily deal with music data in any language simply by preparing a phoneme dictionary. In addition, since analysis of acoustic signals and speech recognition are not performed, processing can be performed at a higher speed than the conventional method. Moreover, the timing information for displaying the lyrics can be given for any musical composition data.

  The best mode for carrying out the present invention will be described below with reference to preferred embodiments.

Embodiment 1.
The lyric data display apparatus 1 of Embodiment 1 based on embodiment of this invention is demonstrated.

  FIG. 1 is a block diagram showing a configuration example of a lyrics data display device 1 based on Embodiment 1 of the present invention.

  In FIG. 1, the lyrics data display device 1 includes an input unit 10, a storage unit 20, a control unit 30, a display unit 40, and a playback unit 50. The storage unit 20 has a music data storage unit 21, a singing time data storage unit 22, a lyrics data storage unit 23, a phoneme dictionary storage unit 24, and a mora number calculation rule storage unit 25 for each function. The control unit 30 includes a phoneme data conversion unit 31, a mora number calculation unit 32, a mora speed calculation unit 33, a display timing setting unit 34, and a display control unit 35.

  The input unit 10 is an input device that can perform selection and determination operations on the lyrics data display device.

  FIG. 2 shows an example of the input unit 10. In FIG. 2, as the input unit 10, a cross key 1000 for performing a selection operation and a determination button 1001 for performing a determination operation are shown.

  The storage unit 20 is a large-capacity recording medium that can be accessed at high speed, such as a hard disk or a flash memory. Hereinafter, the storage unit 20 will be described separately for each function.

  The music data storage unit 21 stores digital data that is the music data main body and music data ID that is an identifier for uniquely identifying the digital data of each music data. The music data ID may be a numerical value such as 1, 2, 3, or a URI (Uniform Resource Identifier). Of course, the music data storage unit 21 may be provided outside the apparatus.

  The singing time data storage unit 22 stores singing time data that is a start time and an ending time of a singing section for each singing section that is an existing section of singing data in the music data. The singing time data of the start time and the ending time of one or more sections (singing section) where the singing data which is the vocal sound of the music data associated with is present is stored. Singing section data is stored by pairing the start time and end time of the singing section.

  In FIG. 3, an example of the format of the singing time data stored in the singing time data storage part 22 is shown. In the example of FIG. 3, the music data ID, the singing section number, and the start time and end time of each singing section are stored. The song data ID “ID1” has two singing sections. The singing section number 1 has a start time of 10 seconds and an end time of 30 seconds. Singing section number 2 has a start time of 50 seconds and an end time of 1 minute 20 seconds. The singing time data storage unit 22 may store a section without vocal sound such as a prelude or an interlude instead of a section with vocal sound. In that case, sections other than the section without vocal voice are converted into singing sections.

  The lyric data storage unit 23 stores the lyric data of the music data described as text data for each singing section, and stores the tune data ID and the lyric data for each singing section of the corresponding music data. .

  FIG. 4 shows an example of the lyrics data stored in the lyrics data storage unit 23. In the example of FIG. 4, the music data ID and the text data of the lyrics including the line feed delimiter, which is obtained by dividing the music data for each singing section, are stored. There are two song sections of the song data with the song data ID “ID1”, and lyrics text data is stored for each song section. Note that Japanese lyrics data is stored in advance in hiragana.

  The phoneme dictionary storage unit 24 stores a phoneme dictionary for converting a character or a word into phoneme data which is a minimum speech unit that has a function of distinguishing the meaning of the character or word. A phoneme is the smallest speech unit that works to distinguish the meaning of a word. As an example in Japanese, “Aki” is composed of a combination of consonant and vowel “aki”, and these / a /, / k /, / i / are phonemes. If you change / k / to / s /, it will be pronounced “ashi” and the meaning of the word will change.

  FIG. 5 shows an example of a phoneme dictionary for English. In the example of FIG. 5, English words are stored on the left side, and phonemes corresponding to the words are stored on the right side. In English, a word is a word and stores phonemes for as many words as possible. In the case of Japanese, the phoneme does not change depending on the word structure, so the phoneme of hiragana is stored instead of the word.

  FIG. 6 shows an example of a phoneme dictionary for Japanese. In the example of FIG. 6, each phoneme is shown in Roman letters in the order of 50 notes “Aiueoka ...”.

  The mora number calculation rule storage unit 25 stores a mora number calculation rule for calculating the number of mora, which is a segmental unit of a sound having a certain temporal length in pronunciation, that is, a mora number, from a phoneme string. A mora is a unit used in phonology, and is a segmental unit of sound having a certain length of time. For example, “Kotatsu” has three sounds having a certain length of time, so the number of mora is three. “Kyatsu” is 3 characters, but the roar “ya” does not have a time length, so the number of mora is counted as 3 as with “Kotatsu”. The mora number calculation rule storage unit 25 will be described in detail in the mora number calculation unit 32 described later. The above is the description of the storage unit 20.

  The control unit 30 includes a CPU and a memory (not shown), and controls the entire lyrics data display device 1 when the CPU executes a program for displaying lyrics displayed on the memory. Hereinafter, the control unit 30 will be described separately for each function.

  The phoneme data conversion unit 31 reads the lyric data corresponding to the designated music data, refers to the phoneme dictionary storage unit 24, and converts the lyric data into phoneme data that is a phoneme sequence that is a sequence of phonemes.

  The operation flow of the phoneme data conversion unit 31 will be described with reference to FIG.

  First, when the music data ID is input, the phoneme data conversion unit 31 refers to the lyrics data storage unit 23 and acquires the lyrics data corresponding to the music data ID (step S310). Next, the phoneme data conversion unit 31 extracts words that can be converted into phoneme data from the unprocessed portion in the acquired lyrics data (step S311). For example, if it is Japanese, it corresponds to one hiragana, and if it is English, a word corresponds to a word that can be converted into phoneme data. Next, the phoneme data conversion unit 31 converts the word extracted in step S311 into phoneme data with reference to the phoneme dictionary storage unit 24 (step S312). Next, the phoneme data conversion unit 31 determines whether or not all lyrics have been converted into phoneme data (step S313). If unprocessed words remain, the process returns to step S311 to repeat the process. Otherwise, the phoneme data conversion unit 31 ends the process.

  When a character corresponding to a line break is extracted in step S311, the phoneme data converter 31 outputs it as a line break to the phoneme string as it is.

  Here, a specific example of processing the English sentence “An apple a day keeps the doctor away” will be described. In this case, the phoneme data conversion unit 31 first extracts an which is the first word that can be converted into phoneme data, refers to the phoneme dictionary storage unit 24, and converts it into phoneme data. Since there are still words that have not been converted to phoneme data, the next word (apple) is taken out and converted to phoneme data. FIG. 8 shows a phoneme string obtained by repeating the above process and converting all words into the phoneme data.

  The mora number calculation unit 32 calculates the mora number of the phoneme string created by the phoneme data conversion unit 31. The number of mora is calculated in units of lines. Hereinafter, the number of mora per line is referred to as the number of line mora. The reason for using mora instead of phonemes is that phonemes do not contain time length information.

  The flow of mora number calculation processing by the mora number calculation unit 32 will be described with reference to the flowchart of FIG.

  First, the mora number calculation unit 32 extracts one line from which the mora number has not yet been calculated from the phoneme string (step S320).

  Next, the mora number calculation unit 32 converts the phoneme string extracted in step S320 into a vowel symbol (V) and a consonant symbol (C), and creates a VC string (step S321). The mora number calculation unit 32 has a conversion table inside, and converts the phoneme to V if the phoneme is a vowel, and converts the phoneme to C if the phoneme is a consonant symbol. In this conversion, if the vowel is a double vowel, it is converted into two Vs, and if it is a triple vowel, it is converted into three Vs. Also, semi-vowels (sounds such as “Ai ぅ e ぅ yayu” in Japanese) are not converted to V or C. In addition, the prompt sound (“t” in Japanese), repellent sound (“n” in Japanese), and long sound (“−” in Japanese) are converted to V.

  In FIG. 10, an example of the conversion table which the mora number calculation part 32 has inside is shown. FIG. 10 is a table for converting phonemes shown in the left column to VCs in the right column.

  FIG. 11 shows an example of conversion from a phoneme string to a VC string. (A) in FIG. 11 is a table showing the phoneme sequence and VC sequence of “An apple a day keeps the doctor away”, and (B) in FIG. 11 is a phoneme sequence of “Irohaniho Chiriru” And the VC column.

  Next, the mora number calculation unit 32 extracts a partial VC sequence from the VC sequence created in step S321 (step S322). In other words, the mora number calculation unit 32 extracts from the original VC sequence to the first found V from the reading position p as a partial VC sequence. The initial value of the reading position p is the first character of the VC string, and after the partial VC string is extracted, the reading position p is set to the position next to the found V. If V is not found up to the end of the VC column, the part from the reading position p to the end of the VC column is taken out as a partial VC column. Further, when extracting the partial VC string, the blank characters between the words are ignored. The reason for this is that, for example, in an English language where the end of a word is a consonant, when “an apple” is actually pronounced, between an and apple, It is because it has.

  As a specific example of extracting a partial VC string, when processing is performed on the VC string shown in FIG. 11A, when the reading position p is the first character of the VC string, the first V is one character. Since it appears in the eyes, the extracted partial VC string is “V”. When the reading position p is the second character, the extracted partial VC string is “CV”.

  Next, the mora number calculation unit 32 acquires the mora number from the partial VC sequence and adds it to the total mora number (step S323). That is, the mora number calculation unit 32 refers to the mora number calculation rule storage unit 25, acquires the number of mora that matches the partial VC sequence, and adds it to the total number of mora. That is, if the total number of mora for the nth VC column of the phoneme column is Mn and the number of mora acquired from the partial VC column is m, the new total mora number is Mn + m.

  FIG. 12 shows an example of the mora number calculation rule storage unit 25. When creating a rule to be stored in the mora number calculation rule storage unit 25, the rule is created in consideration of prosodic features. For example, in the example of FIG. 12, “CV” and “V” each have a mora number of 1, but the combination of two “CV” also has a mora number of 1 is that consonants + vowels are actually pronounced. This is because the pronunciation length is based on the property of having the same length as “C” and “V”.

  Next, the mora number calculation unit 32 determines whether or not all the extraction of the partial VC sequence from the VC sequence has been completed (step S324). If not completed yet, the process returns to step S321, and the process is repeated. If not, the process proceeds to step S325.

  When the extraction of the partial VC strings from the VC string is completed in step S324, the mora number calculation unit 32 determines whether there is any phoneme string row for which the mora number is not calculated (step S325). If the processing of all phoneme strings has been completed, the mora number calculation unit 32 ends the processing. Otherwise, the process returns to step S320.

  FIG. 13 shows a table in which the example of FIG. 11A is divided into partial VC columns and the respective mora numbers are shown. When all the mora numbers shown in FIG. 13 are added, it becomes 16, and this value is the number of row mora. The above is the description of the processing of the mora number calculation unit 32.

  The mora speed calculator 33 calculates the mora speed for each singing section based on the number of row mora calculated by the mora number calculator 32.

  The operation flow of the mora speed calculation unit 33 will be described with reference to the flowchart of FIG.

  First, the mora speed calculation unit 33 refers to the lyrics data storage unit 23 and acquires the start line number and the end line number of the lyrics corresponding to the singing section number i (step S330).

  Next, the total number of mora in the section of the acquired start line number and end line number is calculated from the number of line mora obtained for each lyrics line calculated by the mora number calculation unit 32 (step S331). The number of mora in the singing section obtained here is called the section mora number.

  Next, the time length of the singing section is acquired (step S332). The mora speed calculation unit 33 refers to the singing time data storage unit 22 and acquires a start time and an end time corresponding to the singing section i. If the acquired start time is ts and the end time is te, the time length T is

で求められる。

Is required.

  Next, the mora speed is calculated based on the number of section mora obtained in step S331 and the time length of the singing section obtained in step S332 (step S333). If the number of section mora of the singing section i is Mi and the time length of the singing section i is Ti, the mora speed Vi of the singing section i is

で求められる。

Is required.

  Next, it is determined whether or not the mora speed has been calculated for all singing sections. When the mora speed is calculated for all the singing sections, the mora speed calculation unit 33 ends the process. Otherwise, the singing section i is changed to i + 1, and the process returns to step S330 and continues.

  Here, in the example of FIG. 3 and FIG. 4, the specific example which calculates the mora speed of the song section 1 about the music of music data ID ID1 is shown. Since the end time is 30 seconds and the start time is 10 seconds, the time length T1 of the singing section 1 is calculated as 30-10 = 20 seconds. Assuming that the section mora number M1 of the singing section 1 is 28, the mora speed V1 is calculated as 28/20 and is obtained as 1.4. The above is the description of the mora speed calculation unit 33.

  The display timing setting unit 34 sets the display timing for each lyric line during reproduction based on the mora speed for each singing section obtained by the mora speed calculation unit 33. Assuming that the mora speed of the song section i is Vi and the number of line mora of the song line i in the song section i is Mn, the delay time Dn of the song line i from the song section i start time ts is:

で求めることができ、歌詞行ｎの表示開始時間Ｔｎは、Ｔｎ＝ｔｓ＋Ｄｎで求められる。

The display start time Tn of the lyrics line n can be obtained by Tn = ts + Dn.

  The above-described processing is repeated for all singing sections and all lyric lines, and the lyric line display start time Tn is calculated.

  A specific example in which the display start time is calculated based on the mora speed V1 = 1.4 used in the above-described example of the mora speed calculation unit 33 will be described. First, the display start time of the first lyric line in the song section 1 of the song data ID “ID1” shown in FIG. 4 is equal to the start time of the song section 1, and is 10 seconds. Next, for the second lyric line, the number of line mora in the lyric line 1 is 16, so the delay time D2 is

D2 = M1 / V1 + D1 = 16 / 1.4 + 0≈11.4
The display start time of the lyrics line 2 is calculated as 10 + 11.4 = 21.4 seconds.

  The display control unit 35 displays lyrics according to the reproduction time of the music data. The display control unit 35 checks the reproduction position of the music data being reproduced at regular intervals. Based on the music data playback position at the time of the check, the display start time of the lyrics set by the display timing setting unit 34 is referred to, and when the display start time of the next lyrics is exceeded, the lyrics are displayed. The above is the description of the control unit 30.

  The display unit 40 is a display device such as a display and displays lyrics.

  The playback unit 50 is a playback device such as a music data player, a speaker, or headphones.

  Next, the overall operation of the lyrics data display device 1 in this embodiment will be described with reference to the flowchart of FIG.

  When the lyrics data display device is activated, first, the control unit 30 accepts input of music data to be reproduced (step S10).

  FIG. 16 shows an example of a user interface for performing a music data input operation. In FIG. 16, a list of reproducible music data is displayed on the display unit 40, and the user moves the cursor 1010 via the input unit 10 to determine music data to be played back.

  Next, the control unit 30 acquires the lyrics data of the music data to be reproduced from the lyrics data storage unit 23 from the information input in step S10 (step S20).

  Next, the control unit 30 sets the lyrics display timing (step S30).

  The processing flow of the display timing setting operation will be described with reference to the flowchart of FIG.

  First, the phoneme data conversion unit 31 converts the text data of the lyrics into a phoneme and creates a phoneme string (step S3000). The operation flow of the phoneme data converter 31 is as shown in FIG.

  Next, based on the phoneme sequence created in step S3000, the mora number calculation unit 32 calculates the mora number of the lyrics (step S3010). The operation flow of the mora number calculation unit 32 is as shown in FIG.

  Next, based on the number of mora calculated in step S3010, the mora speed calculation unit 33 calculates the mora speed for each singing section (step S3020). The operation flow of the mora speed calculator 33 is as shown in FIG.

  Next, based on the mora speed for each singing section calculated in step S3020, the display timing setting unit 34 sets the display start timing for each lyrics line (step S3030). The operation of the display timing setting unit 34 is as described above. This completes the description of the lyrics data synchronous display control process.

  Returning to the description of the flowchart of FIG. 15, the control unit 30 reproduces the music data instructed in step S10 (step S40).

  Next, the display control unit 35 determines whether or not the reproduction position of the music data being reproduced has exceeded the display start time of the lyrics line to be displayed next (step S50). If so, the process proceeds to step S60, and if not, the process proceeds to step S70.

  If it is determined that the result is true in step S50, the control unit 30 causes the display unit 40 to display the lyrics line to be displayed next (step S60). An example is shown in FIG. The lyrics 1020 shown in the display unit 40 are the lyrics displayed based on the reproduction time of the music data.

  When it is determined to be false in step S50, or when the process of step S60 is completed, the control unit 30 determines whether or not the reproduction is completed (step S70). If the reproduction has not ended, the process returns to step S50. If the reproduction has ended, the lyrics data display device 1 ends the process. The above is the description of the operation of the lyrics data display device 1.

  As described above in detail, according to the lyric data display device 1 of the first embodiment, it is possible to give the lyric data display start timing information to the lyric data without the synchronization information, and the user can perform special work. Even if not performed, an appropriate lyrics line can be displayed in accordance with the music data to be reproduced.

  In addition, although the example of the mora number calculation rule storage unit 25 in this embodiment is one regardless of the language, it may be prepared for each language such as Japanese, English, German, French, and the like.

  The storage unit 20 may be an external device connected to the lyrics data display device 1 via a network.

  In addition, with respect to music data in any language, it is possible to display an appropriate lyric line in accordance with the music data to be reproduced.

  In addition, since analysis of acoustic signals and voice recognition are not performed, processing can be performed at a higher speed than the conventional method.

Embodiment 2.
Next, the lyrics data display device 2 of Embodiment 2 based on the embodiment of the present invention will be described. Kanji often appears in Japanese lyrics, but the reading of kanji is not only read aloud and kunomi, but also changes depending on the sending kana and usage, and there is no delimiter between words. It is difficult to convert to phoneme data only with a dictionary that contains a large number of words. Therefore, according to the lyrics data display device 2 of the second embodiment, the lyrics can be synchronized even with Japanese lyrics including kanji.

  FIG. 19 is a block diagram illustrating a configuration example of the lyrics data display device 2 according to the second embodiment of the present invention.

  19, in the lyrics data display device 2 of the second embodiment, a kanji dictionary storage unit 26 is added to the storage unit 20 of the lyrics data display device 1 shown in FIG. 2, and a hiragana conversion unit 36 is added to the control unit 30. ing. In the second embodiment, only the kanji dictionary storage unit 26 and the hiragana conversion unit 36 will be described, and the other units are the same as those of the lyrics data display device 1 and will not be described.

  The kanji dictionary storage unit 26 stores words expressed in kanji, reading kana, and parts of speech.

  FIG. 20 shows an example of the format of the Chinese character dictionary storage unit 26. In the example of FIG. 20, “company” stores only nouns and the reading pseudonym is “Kaisha”. In the case of “call”, the reading kana in the case of a noun is “sho”, but in the case of a kabana, for example, the reading kana in the case of “write” is “ka”.

  The hiragana conversion unit 36 converts kanji into hiragana. When a sentence is input, the hiragana conversion unit 36 first performs morphological analysis, which is a known technique, and decomposes the sentence into words including part-of-speech information. Next, based on the decomposed word and the part of speech, the Kanji dictionary storage unit 26 is referred to convert it to Hiragana. This operation is performed for all input sentences.

  Here, a specific example in which the sentence “she buys a cake” is converted into hiragana by the hiragana conversion unit 36 will be shown.

  FIG. 21 shows an example in which “she buys a cake” is converted into a morpheme. When each word in FIG. 21 is converted into a hiragana character with reference to the Kanji dictionary storage unit 26, “Kanojo Kakeki” is obtained.

  Next, an operation example of the lyrics data display device 2 in Embodiment 2 will be described with reference to the flowchart of FIG.

  In the flowchart of FIG. 22, a hiragana conversion process (step S25) is further added to the flowchart (FIG. 15) of the lyrics data display device 1 in the first embodiment. In the present embodiment, the hiragana conversion process is described, and the other processes are the same as those in the first embodiment, and thus the description thereof is omitted.

  When the lyric data is acquired in step S20, the hiragana conversion unit 36 converts the acquired lyric data into hiragana. The operation of the hiragana conversion unit 36 is as described above. In step S30, the lyrics data synchronous display control process is performed using the lyrics data of hiragana converted in step S20. Other processes are the same as those in the first embodiment. The above is the description of the lyrics data display device 2.

  Therefore, according to the lyric data display device 2 of the second embodiment, the lyric synchronization process can be performed even for lyric data including kanji.

  Although the second embodiment deals with kanji in Japanese, it can be similarly applied to languages in which reading changes depending on usage, such as kanji.

Embodiment 3.
The lyric data display apparatus 3 of Embodiment 3 based on embodiment of this invention is demonstrated. According to the lyric data display device 3, even if the lyric data is not divided and stored in the singing section, the synchronization process can be performed.

  FIG. 23 is a block diagram showing a configuration example of the lyrics data display device 3 based on the embodiment of the present invention.

  23, in the lyrics data display device 3, a lyrics line information storage unit 27 is added to the storage unit 20 of the lyrics data display device 1 shown in FIG. 2, and a singing section division unit 37 is added to the control unit 30, respectively. . Further, since the configuration of the lyrics data storage unit 23 of the storage unit 20 is different from the mora speed calculation unit 33 of the control unit 30, in the lyrics data display device 3, the lyrics data storage unit 23A and the mora speed calculation unit 33A are used. . In the third embodiment, only the lyrics data storage unit 23A, the lyrics line information storage unit 27, and the singing section division unit 37 will be described, and the other units are the same as the lyrics data display device 1 and will be described. Is omitted.

  The lyrics data storage unit 23A stores the song data ID and the lyrics of the song data corresponding to the song data ID. The lyric data storage unit 23 of the lyric data display device 1 stores information on the singing section of the music data, but the lyric data storage unit 23A of the lyric data display apparatus 3 does not store the information on the singing section.

  FIG. 24 shows an example of the format of the lyrics data stored in the lyrics data storage unit 23A.

  The example shown in FIG. 24 is obtained by removing the singing section number, which is information related to the singing section, from the example of the format shown in FIG.

  The lyric line information storage unit 27 stores the start line number and the end line number of the lyrics for each singing section calculated by the singing section dividing unit 37. The stored information is given by the singing section dividing unit 37 described later.

  FIG. 25 shows an example of the format of the lyrics line information stored in the lyrics line information storage unit 27. In FIG. 25, for example, for the singing section 1, the start line number is 1 and the end line number is 2.

  The mora speed calculator 33A calculates the mora speed for each singing section based on the number of row mora calculated by the mora number calculator 32. The mora speed calculation unit 33A and the mora speed calculation unit 33 differ in the process of step S330 shown in FIG. In step S330 in the mora speed calculation unit 33, the lyrics data storage unit 23 is referred to obtain the lyrics start line number and end line number corresponding to the singing section number. Since the above information is not stored in the lyric data storage unit 23, the lyric line information storage unit 27 is referred to obtain the start line number and the end line number for each singing section of the lyrics. The other processes are the same as those of the mora speed calculation unit 33, and thus description thereof is omitted.

  The singing section dividing unit 37 divides lyrics data that is not divided into singing sections into singing sections. The flow of operation of the singing section dividing unit 37 will be described with reference to the flowchart of FIG.

  First, the singing section dividing unit 37 calculates the playback time of all singing sections (step S370). The singing section division unit 37 refers to the singing time data storage unit 37 and totals the reproduction times of all the singing sections. Note that the playback time is obtained by subtracting the start time from the end time.

  Next, the song section division unit 37 calculates the average mora speed of the song (step S371). The average mora speed is obtained by dividing the total reproduction time of the song calculated in step S370 from the total number of mora in each lyrics line. The number of row mora is calculated by the mora number calculation unit 32 as described above.

  Next, the singing section dividing unit 37 divides the lyrics into each singing section (step S372). Assuming that the playback time of the song section j is tj, the average mora speed obtained in step S371 is v, and the number of line mora of the lyrics line k is Mk, the expected section mora number mj of the song section j is

で求められる。この値を用いて、

Is required. Using this value,

を満たす、最大のｎｊを求める（ｎｊは整数、ｊは自然数）。このｎｊが、歌唱区間ｊに含まれる歌詞行番号ｎｊとなる。なお、数式５に示されたε０は、予想モーラ数に対して許容される誤差範囲を示す０以上の整数である。この値は固定値でも良いし、

The maximum nj that satisfies the above is obtained (nj is an integer, j is a natural number). This nj becomes the lyrics line number nj included in the singing section j. Note that ε0 shown in Equation 5 is an integer of 0 or more indicating an error range allowed for the expected number of mora. This value can be a fixed value,

のような数式であっても良い。以上の計算を、全ての歌唱区間について行う。

A mathematical expression such as The above calculation is performed for all singing sections.

  Next, the song section division unit 37 stores data in the lyrics line information storage unit 27 (step S373). The starting line number of the singing section j is nj, and the ending line number is stored in the lyrics line information storage unit 27 as (nk) -1 (k = j + 1).

  By the way, in the lyrics, there is data in which blank lines are provided in order to make the lyrics easy to see. In such a case, there is a high possibility that an interlude is inserted at a location corresponding to a blank line. Therefore, the singing section division unit 37 can also use this information. Specifically, a lyric line with no blank line in between is treated as one lyric line.

  Taking the lyrics of ID1 in FIG. 24 as an example, two lines until a blank line appears are collected, that is, “An apple a day keeps the doctor away Time flies like an arrow” is treated as one lyrics line. Therefore, this new row mora number M is a numerical value obtained by summing the number of row mora for two rows. The above is the description of the singing section dividing unit 37.

  Next, an operation example of the lyrics data display device 3 in Embodiment 3 will be described.

  The operation of the lyric data display device 3 is the same as the operation flow of the lyric data display device 1 in the first embodiment shown in FIG. 15, but the detailed processing operation of the synchronous display control processing of the lyric data in step S30 is as follows. Different.

  FIG. 27 is a flowchart illustrating an example of the operation of the lyrics data synchronous display control process of the lyrics data display device 3 according to the third embodiment. Note that the phoneme conversion process (step S3000) and the mora number calculation process (step S3010) have the same processing contents as the lyric data display device 1 in the first embodiment, and thus description thereof is omitted.

  When the calculation of the number of line mora is completed, the singing section dividing unit 37 calculates a starting lyrics line and an ending lyrics line for each singing section, and stores them in the lyrics line information storage unit 27 (step S3015). The processing of the singing section dividing unit 37 is as described above.

  After setting a value in the lyrics line information storage unit 27, the mora speed calculation unit 33A calculates the mora speed for each singing section (step S3020A). The process of the mora speed calculation unit 33A is as described above.

  The display timing setting process (step S3030) for setting the display start timing for each lyric line from the mora speed for each singing section is the same as the lyric data display device 1 in the first embodiment, and thus the description thereof is omitted. To do. The operation of the lyrics data display device 3 has been described above.

  As described above in detail, according to the lyrics data display device 3 in Embodiment 3, even if the lyrics data is not divided into singing sections, appropriate lyrics can be displayed during the reproduction of the music data. .

 In the first to third embodiments, the lyric data display device according to the present invention has been described as being configured by hardware as shown in the block diagrams of FIG. 1, FIG. 19, and FIG. 23. However, the present invention is not limited to this. The lyric data display program for executing the functions of the lyric data display apparatuses according to the first to third embodiments and the CPU for executing the lyric data display program may be configured by software. Of course.

1 is a block diagram illustrating a configuration of a lyrics data display device 1 according to Embodiment 1. FIG. It is a figure which shows an example of the input part in Embodiment 1-3. It is a figure which shows an example of the singing time data in Embodiments 1-3. It is a figure which shows an example of the lyric data in this invention Embodiment 1,2. It is a figure which shows an example of the phoneme dictionary of English in Embodiment 1-3. It is a figure which shows an example of the Japanese phoneme dictionary in Embodiment 1-3. It is a flowchart which shows the flow of a process of the phoneme data conversion part in Embodiment 1-3. It is a figure which shows an example which converted the English sentence into the phoneme string. It is a flowchart which shows the flow of a process of the mora number calculation part in Embodiment 1-3. It is a figure which shows an example of the conversion table for converting into a VC row | line | column from the phoneme in the mora number calculation part in Embodiment 1-3. It is a figure which shows an example which converted the phoneme string into the VC string. It is a figure which shows an example of the mora number calculation rule in Embodiment 1-3. It is a figure which shows an example converted into the number of mora from a VC row | line | column. It is a flowchart which shows the flow of a process of the mora speed calculation part in Embodiment 1-3. 6 is a flowchart illustrating an example of the operation of the lyrics data display device 1 according to the first embodiment. It is a figure which shows an example of the user interface for performing the input operation of music data. It is a flowchart which shows an example of the synchronous display control processing operation | movement of the lyrics data in Embodiment 1, 2 of this invention. It is a figure which shows an example which displayed the lyrics on the display part. It is a block diagram which shows the structure of the lyric data display apparatus 2 in Embodiment 2. FIG. It is a figure which shows an example of the Chinese character dictionary in Embodiment 2. It is a figure which shows an example which performed the morphological analysis from the sentence to the word and the part of speech. 10 is a flowchart illustrating an example of the operation of the lyrics data display device 2 according to the second embodiment. It is a block diagram which shows the structure of the lyric data display apparatus 3 in Embodiment 3. FIG. It is a figure which shows an example of the lyric data in Embodiment 3. It is a figure which shows an example of the lyric line information in Embodiment 3. It is a figure which shows an example of operation | movement of the song section division part in Embodiment 3. 15 is a flowchart illustrating an example of an operation of synchronous display control processing of lyrics data in the third embodiment.

Explanation of symbols

1, 2, 3 Lyrics data display device 10 Input unit 20 Storage unit 21 Music data storage unit 22 Singing time data storage unit 23, 23A Lyric data storage unit 24 Phoneme dictionary storage unit 25 Mora number calculation rule storage unit 26 Kanji dictionary storage unit 27 Lyric line information storage unit 30 Control unit 31 Phoneme data conversion unit 32 Mora number calculation unit 33, 33A Mora speed calculation unit 34 Display timing setting unit 35 Display control unit 36 Hiragana conversion unit 37 Singing segment division unit 40 Display unit 50 Playback unit

Claims (7)

A singing time data storage unit that stores singing time data that is a start time and an end time of the singing section for each singing section that is an existing section of singing data in the music data;
A lyric data storage unit for storing lyric data of the music data described as text data for each singing section;
A phoneme dictionary storage unit that stores a phoneme dictionary for converting characters or words into phoneme data, which is a minimum speech unit that works to distinguish the meaning of the character or word;
From the phoneme data, a mora number calculation rule storage unit that stores a mora number calculation rule for calculating the number of mora that is a segmental unit of a sound having a certain temporal length in pronunciation;
Reading the lyrics data from the lyrics data storage unit, referring to the phoneme dictionary storage unit, and converting the lyrics data into the phoneme data;
Read the phoneme data created by the phoneme data conversion unit with reference to the mora number calculation rule, and the number of mora in the singing section and the number of mora in each row of the singing section A mora number calculation unit for calculating the number of row mora;
The singing time data is read from the singing time data storage unit, the time length of the singing section is calculated, and the section mora number calculated by the mora number calculating unit is obtained by dividing by the time length. A mora speed calculator for calculating a mora speed for each singing section;
A display timing calculation unit for calculating a display timing for each line of the lyrics data based on the mora speed calculated by the mora speed calculation unit and the line mora number calculated by the mora number calculation unit;
Based on the display timing calculated by the display timing calculation unit, a display control unit for displaying the lyrics data in synchronization with the reproduction of the music data;
A lyric data display device. A singing time data storage unit that stores singing time data that is a start time and an end time of the singing section for each singing section that is an existing section of singing data in the music data;
A lyrics data storage unit for storing lyrics data of the music data described as text data;
A phoneme dictionary storage unit that stores a phoneme dictionary for converting characters or words into phoneme data, which is a minimum speech unit that works to distinguish the meaning of the character or word;
From the phoneme data, a mora number calculation rule storage unit that stores a mora number calculation rule for calculating the number of mora that is a segmental unit of a sound having a certain temporal length in pronunciation;
Reading the lyrics data from the lyrics data storage unit, referring to the phoneme dictionary storage unit, and converting the lyrics data into the phoneme data;
Read the phoneme data created by the phoneme data conversion unit with reference to the mora number calculation rule, and the total number of mora that is the number of mora of the entire lyrics data and the number of mora for each row of the lyrics data A mora number calculating unit for calculating the number of row mora,
The singing time data is read from the singing time data storage unit, the time length of each singing section and the total time length that is the total value thereof are calculated, and the total number of mora calculated by the mora number calculating unit is The average mora speed obtained by dividing by the total time length is calculated, and the estimated mora number of each singing section obtained by dividing the time length of each singing section by the average mora speed and the number of row mora Based on the singing section dividing unit for making the lyrics data correspond to each singing section for each line,
The number of mora for each singing section is calculated based on the correspondence information between the lyric data obtained by the singing section dividing unit and the singing section and the number of row mora calculated by the mora number calculating unit. And a mora speed calculation unit for calculating a mora speed for each singing section, which is obtained by dividing the time length of the singing section calculated by the singing section dividing unit,
A display timing calculation unit for calculating a display timing for each line of the lyrics data based on the mora speed calculated by the mora speed calculation unit and the line mora number calculated by the mora number calculation unit;
Based on the display timing calculated by the display timing calculation unit, a display control unit for displaying the lyrics data in synchronization with the reproduction of the music data;
A lyric data display device. In the lyric data display device according to claim 1 or 2,
further,
A kanji dictionary storage unit for storing one or more hiragana corresponding to kanji and how to read the kanji,
A hiragana conversion unit that converts kanji in the lyrics data acquired from the lyrics data storage unit into hiragana with reference to the kanji dictionary storage unit,
The phoneme data converter is
A lyric data display device that refers to the phoneme dictionary storage unit, converts the lyric data converted into hiragana by the hiragana conversion unit into the phoneme data, and creates a phoneme string. The lyrics data of the music data described as text data composed of one or more lines corresponding to the singing section that is the singing data existing section in the music data is read, and the meaning of the word or word is distinguished. Creating a phoneme time-series data obtained by converting the lyric data into the phoneme data with reference to a phoneme dictionary for converting to phoneme data that is a minimum speech unit that has a function to perform;
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the number of section mora that is the number of mora in the singing section and each row of the singing section Calculating the number of row mora which is the number of mora of
Calculating the mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
Calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
A method for displaying lyric data. To read lyric data of music data described as text data composed of one or more lines, and to convert characters or words into phoneme data, which is the smallest speech unit that works to distinguish the meaning of words Creating a phoneme time-series data obtained by converting the lyrics data into the phoneme data, referring to the phoneme dictionary of
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the total number of mora that is the number of mora of the entire music data and the lyrics data Calculating the number of row mora, which is the number of mora in each row;
A step of calculating an average mora speed obtained by dividing the total number of mora by a total value of time lengths of each singing section which is an existing section of singing data in the music data;
Calculating an estimated number of mora for each song section obtained by dividing the time length of each song section by the average mora speed;
Creating correspondence information between each line of the lyrics data and each song section based on the estimated number of mora and the number of line mora;
Based on the correspondence information and the number of row mora, calculating a section mora number that is the number of mora for each singing section;
A mora speed calculating step for calculating a mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
A timing calculating step for calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
A method for displaying lyric data. The lyrics data of the music data described as text data composed of one or more lines corresponding to the singing section that is the singing data existing section in the music data is read, and the meaning of the word or word is distinguished. Creating a phoneme time-series data obtained by converting the lyric data into the phoneme data with reference to a phoneme dictionary for converting to phoneme data that is a minimum speech unit that has a function to perform;
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the number of section mora that is the number of mora in the singing section and each row of the singing section Calculating the number of row mora which is the number of mora of
Calculating the mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
Calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
Lyrics data display program that makes a computer execute. To read lyric data of music data described as text data composed of one or more lines, and to convert characters or words into phoneme data, which is the smallest speech unit that works to distinguish the meaning of words Creating a phoneme time-series data obtained by converting the lyrics data into the phoneme data, referring to the phoneme dictionary of
For the phoneme time-series data, calculate the number of mora that is a segmental unit of sound having a certain temporal length in pronunciation, and the total number of mora that is the number of mora of the entire music data and the lyrics data Calculating the number of row mora, which is the number of mora in each row;
A step of calculating an average mora speed obtained by dividing the total number of mora by a total value of time lengths of each singing section which is an existing section of singing data in the music data;
Calculating an estimated number of mora for each song section obtained by dividing the time length of each song section by the average mora speed;
Creating correspondence information between each line of the lyrics data and each song section based on the estimated number of mora and the number of line mora;
Based on the correspondence information and the number of row mora, calculating a section mora number that is the number of mora for each singing section;
A mora speed calculating step for calculating a mora speed for each singing section obtained by dividing the number of section mora by the time length of the singing section;
A timing calculating step for calculating a display timing for each line of the lyrics data based on the mora speed and the number of line mora;
Displaying the lyrics data in synchronization with the reproduction of the music data based on the display timing;
Lyrics data display program that makes a computer execute.

Images