TW201415884A - System for adjusting display time of expressions based on analyzing voice signal and method thereof - Google Patents

Abstract

A system for adjusting display time of expressions based on analyzing a voice signal and a method thereof are provided. By obtaining start time of speech sections in the voice signal after analyzes the voice signal, corresponding each expression of an exposition corresponding with the voice signal to each of the speech sections in sequence, and adjusting display time of each of the expressions to the start time of the speech section corresponding the expression, the system and the method can improve efficiency of adjusting display time of expressions, and can achieve the effect of adjusting display time of each expression to appearance time of voice corresponding the expression.

Description

System and method for synchronizing vocal signals with textual description materials

A phrase display time synchronization system and method thereof, in particular, a system and method for synchronizing a human voice signal with a text description material thereof.

The technical solution of dynamic lyrics can read the lyrics corresponding to the song files when the song files are played, and synchronously display the lyrics represented by the played vocals. In fact, dynamic lyrics are not limited to use in song files. Any multimedia file that is representative of the vocal that is included can use dynamic lyrics to simultaneously display the words represented by the vocals.

As more and more multimedia player software or multimedia players support dynamic lyrics, the owner of the multimedia file gradually hopes that the multimedia player or multimedia player can simultaneously display the vocal being played when playing the audio signal in the multimedia file. The words expressed.

In the text description data for providing the dynamic lyrics, the display time corresponding to each word recorded in the text description data must be included, so that the multimedia playing software or the multimedia player can correspond to each word at the time when the audio signal is played. When the display time is the same, each word is played synchronously.

Although most of the current audio signals have corresponding text description materials to use the dynamic lyrics technical solution, and currently there is a difference between the display time of each word in the text description data and the time when each voice in the audio signal appears, there are also The tool software that modifies the display time recorded in the text description data. However, these tools only provide users with the ability to manually modify the display time of each sentence, or to increase or decrease the display time of all words.

Since there are still some audio signals corresponding to the audio signal, only the words of the audio signal are recorded, and the display time is not recorded. In addition, some audio signals may have multiple versions, and each version may have a vocal time. A little difference, for example, some vocals may be played earlier/late than the time when the words are displayed. If you want to manually add or modify the display time of each phrase, it will take a lot of time and is very inefficient.

In summary, it can be seen that there has been a problem in the prior art that the display time of each word corresponding to the audio signal cannot be efficiently adjusted. Therefore, it is necessary to propose an improved technical means to solve this problem.

In view of the prior art, there is a problem that the display time of the adjustment words is too inefficient, and the present invention discloses a synchronization system and a method for the vocal signals and the text description materials thereof, wherein: the synchronization system of the vocal signals and the text description materials disclosed by the present invention The method includes at least: a loading module for loading an audio signal, an audio signal corresponding to the text description data, and a vocal extraction module for extracting a vocal signal from the audio signal, the vocal signal includes a voice passage, and the paragraph analysis module, It is used to analyze the vocal signal to obtain the start time of the speech passage; the word corresponding module is used to correspond to the words in the speech paragraph and the text description data; the time adjustment module is used to adjust the display time of the words to the corresponding The start time of the voice paragraph.

The method for synchronizing the human voice signal and the text description data disclosed by the present invention comprises the steps of: loading an audio signal, the audio signal corresponding to the text description data; extracting the human voice signal from the audio signal, the human voice signal including the voice passage; analyzing the voice signal To obtain the start time of the phonetic paragraph; The sequence corresponds to the speech passage; the display time of the adjustment phrase is the start time of the corresponding speech segment.

The system and method disclosed in the present invention are different from the prior art in that the present invention obtains the start time of the voice segment in the voice signal by analyzing the voice signal in the audio signal, and displays the text corresponding to the audio signal. After the words in the data are sequentially mapped to the respective speech segments, the display time of each sentence is adjusted to the start time of the corresponding speech segment, so as to solve the problems existing in the prior art, and the time adjustment of the display of the words can be automatically achieved. The technical effect of the time corresponding to the vocal.

The features and embodiments of the present invention will be described in detail below with reference to the drawings and embodiments, which are sufficient to enable those skilled in the art to fully understand the technical means to which the present invention solves the technical problems, and The achievable effects of the present invention.

The invention can detect the human voice signal in the audio signal, and adjust the display time of the words corresponding to each voice segment according to the start time of each voice segment in the voice signal, so that all the words can be played in the corresponding voice segment. The time is displayed without the phenomenon of displaying in advance or delay.

The audio signal of the present invention includes at least a human voice signal, and a human voice, such as a song file, a crosstalk recording file, etc., is generated after being played, but the present invention is not limited thereto, and even the audio signal may be included in the multimedia. In the audio and video files. Among them, the vocal signal may contain one or more voice passages, and each voice passage is separated by a human voice, that is, there is no vocal between the voice passage and the voice passage.

The following is the vocal signal and text description of the invention mentioned in the first figure. The synchronization system architecture diagram of the material illustrates the operation of the system of the present invention. As shown in FIG. 1, the system of the present invention includes a loading module 110, a vocal extraction module 120, a paragraph analysis module 130, a vocabulary correspondence module 150, and a time adjustment module 160.

The loading module 110 is responsible for loading the audio signal. Generally, the loading module 110 loads the audio signal into a memory (not shown) of the device for performing the present invention, but the present invention does not limit. The loading module 110 may load an audio signal from a storage medium (not shown) for executing the apparatus of the present invention, or may load an audio signal by a device external to the apparatus for performing the present invention, and the present invention is not particularly limited.

The audio signal loaded by the loading module 110 has corresponding text description data. For example, when the audio signal is a song file, the text description data is a corresponding lyric file. When the audio signal is a comic recording file, the text description data is For the corresponding subtitle file, when the audio signal is included in the multimedia video file, the text description data is a subtitle file of the multimedia audio and video file containing the audio signal, but the text description of the present invention is not limited to the above.

The caption data corresponding to the audio signal may be stored in the storage medium of the apparatus of the present invention, or may be stored in a device external to the apparatus embodying the present invention, and the present invention is not particularly limited. It is worth mentioning that the audio signal and the corresponding text description data are not necessarily stored in the same device.

In general, when loading the audio signal, the loading module 110 also loads the corresponding text description data into the memory of the apparatus for performing the present invention, but the invention is not limited thereto.

The vocal extraction module 120 is responsible for extracting the vocal signals from the audio signals loaded by the loading module 110. The vocal signal extracted by the vocal module 120 is played for the same time as the audio signal loaded by the loading module 110 is played.

The vocal extraction module 120 can attenuate a specific frequency in the audio signal, for example, attenuating frequencies in the audio signal ranging from 300 Hz to 3000 Hz, so that the non-human frequency will be attenuated, that is, after the audio signal is attenuated The remaining part is mainly the vocal signal. The vocal extraction module 120 can also reverse the left channel of the audio signal, superimpose it on the right channel, and reverse the right channel of the audio signal, and superimpose it on the left channel. Thus, the intermediate signal for eliminating the vocal can be obtained. Then, after the mediation signal of the vocal elimination is reversed, the original audio signal is superimposed, or the original audio signal is reversed and superimposed with the mediation signal, and the signal generated by the superposition is a vocal signal. However, the manner in which the human voice extraction module 120 of the present invention extracts the voice signal from the audio signal is not limited to the above.

The paragraph analysis module 130 is configured to analyze the vocal signals extracted by the vocal extraction module 120 to obtain the start time of each voice segment in the vocal signal. In general, the paragraph analysis module 130 detects the vocal sound in the vocal signal and determines the time point at which the vocal is not sustained (interrupted or stopped) when detecting the vocal sound, and then continues to detect the new vocal, when the paragraph analysis mode When the group 130 detects that the vocal is not sustained, it will treat the continuous vocal sound detected before the vocal interruption or stop as a voice passage, and then detect the new vocal voice, and then detect the new vocal For a new voice passage, the time point at which the paragraph analysis module 130 detects the start of the continuous voice is the start time mentioned in the present invention. However, the manner in which the paragraph analysis module 130 of the present invention obtains each voice segment in the voice signal is not limited to the above.

The vocabulary correspondence module 150 is responsible for establishing a correspondence between each vocabulary in the transcript corresponding to the audio signal and each speech segment obtained by the paragraph analysis module 130, that is, corresponding to each vocal passage in the transcript data. . In general, the phrase correspondence module 150 will arrange the words in the data according to the text description. The sequence and paragraph analysis module 130 obtains the sequence of each voice segment, and sequentially assigns each word segment to each voice segment. However, the manner in which the phrase correspondence module 150 of the present invention corresponds to the word segment and the voice segment is not limited to the above.

When the caption data includes a display time corresponding to each vocabulary, for example, when the caption data is a dynamic lyrics (LRC) file, the vocabulary correspondence module 150 may firstly correspond to each vocabulary. The display time rearranges the words, and then according to the order of the rearranged words, the words are sequentially assigned to the respective speech segments. Among them, it is worth mentioning that some words in the caption data may correspond to two or more display times, and the phrase corresponding module 150 will correspond to two or more words showing the time. The plurality of words are divided into the same number of words as the corresponding display time, and each word corresponds to a different display time, and then each word is arranged according to the display time corresponding to each word.

The time adjustment module 160 is responsible for displaying the correspondence between the words and the speech passages established by the morphological correspondence module 150 and the start time of each speech segment recorded by the paragraph analysis module 130, and displaying the words in the text description data. The time is adjusted to the start time of the corresponding speech segment. The time adjustment module 160 can adjust the display time recorded by the text description data loaded into the memory by the loading module 110, or can be directly used when the text description data is stored in the storage medium of the device for executing the present invention. Adjust the display time recorded in the text description data stored in the storage medium.

In addition, the present invention may further include an additional play module 190, and the play module 190 is responsible for synchronously displaying the words with the same time as the voice segment in the audio signal being played, that is, the time and time when the audio signal is played. Adjustment module When the display time of a certain sentence after the adjustment is the same, the words whose adjusted display time is the same as the time when the audio signal is played are displayed. In some embodiments, the playing module 190 can also play the audio signal loaded by the loading module 110.

Next, the operation system and method of the present invention will be explained with reference to the first embodiment, and please refer to the flowchart of the method for synchronizing the human voice signal and the text description data of the present invention. In this embodiment, it is assumed that the audio signal is a song file, and the corresponding text description data is a lyric file.

When the user downloads the song file (audio signal) to the hard disk of the computer, when using the computer to play the downloaded song file, if the version of the song file is different, the time of displaying the lyrics (sentences) is more than the lyrics in the song file. The time of occurrence of the (speech paragraph) may be slightly different, that is, the display time of the lyrics recorded in the lyric file (text description material) is different from the start time of the lyrics in the song file, resulting in a time for displaying the lyrics (phrases). The lyrics in the song file appear earlier or later, so that the user can use the present invention to adjust the display time of each lyric recorded in the lyrics file.

First, the loading module 110 can load an audio signal (step 210). In this embodiment, the song file is loaded into the computer's memory from the hard disk of the computer. At the same time, it is assumed that the loading module 110 also loads the lyric file corresponding to the song file into the memory of the computer.

After the loading module 110 loads the audio signal (step 210), the vocal extraction module 120 can extract the vocal signal from the audio signal loaded by the loading module 110 (step 220), and then the paragraph analysis module. The vocal signal extracted by the vocal sampling module 120 can be analyzed to obtain the start time of each voice segment in the vocal signal (step 230). In this embodiment, it is assumed that the voice signal contains 12 voices. paragraph.

After the paragraph analysis module 130 obtains the start time of each voice segment in the voice signal, the word correspondence module 150 can sequentially associate each statement in the text description data to each voice segment of the voice signal (step 250). In this embodiment, since the lyrics file (character description material) includes the display time of the lyrics (phrases), the vocabulary correspondence module 150 reorders the lyrics according to the display time of the 12 lyrics recorded in the lyric file. After sorting the lyrics, the 12 lyrics after sorting are sequentially corresponding to the 12 speech passages in the vocal signal, so that each lyrics sequentially corresponds to different speech passages.

In this embodiment, if the lyric file contains only 11 lyrics, but one of the lyrics corresponds to two display times, the vocabulary correspondence module 150 copies the lyrics corresponding to the two display times into the same two lyrics, and The two display times correspond to the same two lyrics that will be copied. In this way, the lyrics file will become 12 lyrics, and each lyrics corresponds to a different display time. Then, the vocabulary correspondence module 150 can reorder the lyrics according to the display time corresponding to each lyric, and sequentially sort the 12 lyrics after the sorting to the 12 speech segments in the vocal signal.

After the word correspondence module 150 sequentially associates each sentence in the text description data to each voice segment of the voice signal (step 250), the time adjustment module 160 can start from each voice segment obtained by the paragraph analysis module 130. At the beginning time, the display time of each sentence in the caption material is adjusted to the start time of the speech segment corresponding to each sentence (step 260). In this embodiment, the time adjustment module 160 adjusts the display time of each sentence recorded in the memory of the computer used by the user to the start time of each voice segment corresponding to each sentence, and also displays the user. The display time of each sentence in the lyric file recorded by the storage medium of the computer used is adjusted to each language The start time of each speech segment corresponding to the sentence. Thus, with the present invention, the display time of each sentence in the lyric file is automatically adjusted to the correct time, and the user does not need to adjust it by himself.

Next, the operation system and method of the present invention will be explained in the second embodiment. Please continue to refer to the method flowchart of "Fig. 2". In this embodiment, it is assumed that the audio signal is also a song file, and the text description material is also a lyric file.

First, the load module 110 can load the song file (audio signal) from the hard disk of the computer into the memory of the computer (step 210). In this embodiment, it is assumed that the loading module 110 also downloads the lyric file corresponding to the song file loaded by the loading module 110 through the network to the lyric server, wherein the lyric file downloaded by the module 110 is loaded. There is no display time for the lyrics.

After the loading module 110 loads the audio signal (step 210), the vocal extraction module 120 can extract the vocal signal from the audio signal loaded by the loading module 110 (step 220), and the paragraph analysis module 130 can The vocal signals extracted by the vocal sampling module 120 are analyzed to obtain the start time of each voice segment in the vocal signal (step 230), and the vocabulary correspondence module 150 can sequentially correspond to the vocals in the text description data. Each speech segment of the signal (step 250).

In this embodiment, since the lyrics file (character description material) does not include the display time of the lyrics (phrases), the vocabulary correspondence module 150 sequentially matches the lyrics according to the order of the lyrics recorded in the lyrics file. Each speech segment in the vocal signal causes each lyric to sequentially correspond to a different speech segment.

After the word correspondence module 150 sequentially associates each sentence in the text description data to each voice segment of the voice signal (step 250), the time adjustment module 160 can adjust the display time of each statement in the text description data to Voice corresponding to each statement The start time of the paragraph (step 260). In this embodiment, since the lyric file re-entered by the loading module 110 does not include the display time, there is no corresponding display time for each lyric recorded in the memory of the computer used by the user, and the time adjustment module 160 adds the start time of the voice segment corresponding to each lyric to the memory of the computer used by the user, so that the display time of each statement recorded in the memory is adjusted from no speech corresponding to each sentence. The start time of the paragraph.

In the above two embodiments, if the play module 190 is included, the play module 190 can play the song file (audio signal) (step 280), and when the song file is played, determine whether the time when the song file is played and whether the memory is played. The display time of each lyric (sentence) recorded in the body (the display time adjusted by the time adjustment module 160) is the same, when the time when the song file is played is the same as the display time of the lyrics recorded in the memory. The playing module 190 can display the lyrics corresponding to the same time as the time when the song file is played, so as to synchronously display the words with the same display time and the same time as the voice segment in the audio signal is played (step 290). Thus, with the present invention, the display time of each sentence in the lyric file is automatically adjusted to the correct time, and the user does not need to adjust it by himself.

In summary, it can be seen that the difference between the present invention and the prior art is that the start time of the voice segment in the voice signal is obtained after analyzing the voice signal in the audio signal, and is in the text description data corresponding to the audio signal. After the words correspond to the respective speech passages in sequence, the technical means for adjusting the display time of each phrase to the start time of the corresponding speech segment can be solved by using a technical means to solve the problem that the display time of the adjustment words in the prior art is too inefficient. The problem, in turn, achieves the technical effect of automatically adjusting the display time of the words to the time when the corresponding vocals appear.

Furthermore, the method for synchronizing the human voice signal of the present invention with the text description data can be It can be implemented in hardware, software or a combination of hardware and software. It can also be implemented in a centralized manner in a computer system or in a distributed manner in which different components are interspersed among several interconnected computer systems.

While the embodiments of the present invention have been described above, the above description is not intended to limit the scope of the invention. Any modification of the form and details of the practice of the present invention, which is a matter of ordinary skill in the art to which the present invention pertains, is a patent protection of the present invention. range. The scope of the invention is to be determined by the scope of the appended claims.

110‧‧‧Loading module

120‧‧‧Voice extraction module

130‧‧‧Paragraph Analysis Module

150‧‧‧word corresponding module

160‧‧‧Time adjustment module

190‧‧‧Playing module

Step 210‧‧‧ Load the audio signal, the audio signal corresponds to the text description, and the text description contains multiple words

Step 220‧‧‧ Extracting the vocal signal from the audio signal, the vocal signal contains multiple Voice passage

Step 230‧‧‧Analyze the vocal signal to get the start time of the utterance

Step 250‧‧‧ Correspondence to the various passages in the textual description

Step 260‧‧‧ The display time of the adjustment statement is the start time of the corresponding speech segment

Step 280‧‧‧Play audio signal

Step 290‧‧‧ Synchronous display shows the same time as the voice passage of the audio signal is played

Figure 1 is a schematic diagram of the synchronization system architecture of the human voice signal and its text description data.

FIG. 2 is a flow chart of a method for synchronizing a human voice signal and a text description data thereof according to the present invention.

Step 210‧‧‧ Load the audio signal, the audio signal corresponds to the text description, and the text description contains multiple words

Step 220‧‧‧ Extracting the vocal signal from the audio signal, the vocal signal contains multiple voice passages

Step 230‧‧‧Analyze the vocal signal to get the start time of the utterance

Step 250‧‧‧ Correspondence to the various passages in the textual description

Step 260‧‧‧ The display time of the adjustment statement is the start time of the corresponding speech segment

Step 280‧‧‧Play audio signal

Step 290‧‧‧ Synchronous display shows the same time as the voice passage of the audio signal is played

Claims (10)

A method for synchronizing a vocal signal with a caption data, the method comprising at least the following steps: loading an audio signal, the audio signal corresponding to a caption data, the caption data comprising a plurality of words; and extracting a human voice from the audio signal a signal, the human voice signal includes a plurality of voice passages; analyzing the voice signal to obtain a start time of each voice segment; sequentially matching each word in the text description data to each of the voice passages; and adjusting each of the words The display time is the start time of the corresponding speech segment. For example, in the method for synchronizing the vocal signal and the caption data described in claim 1, wherein the synchronizing method further comprises playing the step after adjusting the display time of each phrase as the starting time of the corresponding speech segment. The audio signal is displayed in synchronization with the steps of displaying the words with the same time as the audio segment of the audio signal being played. For the method of synchronizing the vocal signal and the caption data as described in claim 1, wherein the step of extracting the vocal signal from the audio signal is to attenuate the specific frequency in the audio signal, or to superimpose the audio in reverse. After the left and right channels of the signal generate an intermediate signal, the audio signal and the intermediate signal are superimposed. For example, the method for synchronizing the vocal signal and the text description material mentioned in the first item of the patent application, wherein the words in the text description data are sequentially corresponding to each The step of the voice paragraph further includes the step of associating each of the words into the respective voice segments according to the arrangement order after the words are arranged according to the display time corresponding to each of the words. The method for synchronizing the vocal signal and the caption data according to item 4 of the patent application scope, wherein the synchronizing method further comprises dividing the corresponding display time according to the step of arranging the words according to the display time corresponding to each word segment. A word is a step that is the same as the number of display times and corresponds to a plurality of words of each display time. A synchronization system of a vocal signal and a text description data, the synchronization system at least comprising: a loading module for loading an audio signal, the audio signal corresponding to a text description material, the text description material comprising a plurality of words; a human voice Extracting a module for extracting a human voice signal from the audio signal, the human voice signal comprising a plurality of voice passages; and a paragraph analysis module for analyzing the voice signal to obtain a start time of each voice segment; The corresponding module is configured to correspond to each of the voice passages and the words in the text description data; and a time adjustment module is configured to adjust the display time of each of the words to the start time of the corresponding voice segment. The synchronization system of the vocal signal and the caption data described in claim 6 wherein the synchronization system further comprises a play module for playing the audio signal, and simultaneously displaying the display time and the audio signal. The speech passages are played with the same words at the same time. The synchronization system of the vocal signal and the text description material described in claim 6 wherein the vocal singer module attenuates the specific frequency in the audio signal or inversely superimposes the left and right channels of the audio signal to generate an intermediary. After the signal, the audio signal and the intervening signal are superimposed in a reverse manner, so that the human voice signal is extracted from the audio signal. For example, the synchronization system of the vocal signal and the text description material described in claim 6 of the patent scope, wherein the vocabulary corresponding module is further configured to arrange the words according to the display time corresponding to each vocabulary, and then the words are arranged according to the arrangement order. Corresponds to each of the speech passages. For example, in the synchronization system of the vocal signal and the text description material described in claim 9, wherein the vocabulary corresponding module is further configured to divide the vocabulary corresponding to the plurality of display times to be the same as the display time and corresponding to each The plurality of words of the display time are arranged according to the display time corresponding to each of the words.