KR20010028053A - Caption data syncronizing method at the Digital Audio Data system - Google Patents

Abstract

PURPOSE: A method of synchronizing digital audio data caption is provided to be capable of performing edition and calling of various caption information based on digital audio data by searching reproduce timing information through extracting and comparing a waveform of a core word and mapping the searched information to digital audio data. CONSTITUTION: If a caption input mode of digital audio data is selected, whether syllable cease arises and whether a display screen separation error exists, are judged. A caption unit is generated in this order of a syllable, a line, a paragraph and a page, and a catalog is displayed. After selecting a caption unit for synchronizing through a frame data mapping process, frame data is generated, and then whether a frame number is automatically designated is judged. If so, a subroutine is performed so as to designate a frame number of a caption unit. If not, a synchronous frame number is designated to the caption unit according to a reproduce content, and then whether a frame number among the caption unit is corrected is judged. In the case where no correction is needed, each frame number is designated according to a syllable unit.

Description

Caption data syncronizing method at the Digital Audio Data system}

The present invention enables the editing and retrieval of various caption information based on digital audio data, optimizes the caption display amount according to the display capability of the liquid crystal display device, and synchronizes digital audio data caption synchronization playback with a simple caption designation algorithm. It is about a method.

Basically, digital audio data files contain only information about voice and music. Caption information that can be defined by a user can be mapped to the digital audio data file.

Digital audio data is composed of a minimum unit whose bit string is called a frame (or AAU: Audio Access Unit). These frame units include the audio portion of all DVD (Digital Versatile Disk) standards and MPEG standards in addition to the MP3 format.

Since the software for mapping caption data to such digital audio data can indicate the position at which the caption is displayed by the frame number, it can be applied to all the digital audio data having a bit string of data configured in units of frames.

When caption data mapped to a digital audio data file is to be output to a display device, for example, a liquid crystal display, how can a user easily find out the time or frame number at which the caption corresponding to the reproduced digital voice or music starts? The problem arises.

Currently, as a caption solution on digital audio data, the characters to be captioned in advance are divided into text data, and when the corresponding music or voice is output, the approximate frame or time information at the point of time when the caption inputted by the time signature is displayed is used. The caption character designation has been developed, but its performance is not expected.

In order to solve the above problem, a technique for mapping caption characters based on digital audio data has been previously filed by the present applicant.

According to this technique, as shown in FIG. 1, when the caption data input mode is selected, in the first step, it is determined whether the size of the caption display screen is determined for caption data input, and if the screen size is not determined, the caption output is performed. After determining the caption display screen size by determining the screen size, it is determined whether the caption is loaded by performing the following steps.

If the caption is not loaded in the above step, if the caption is not loaded, it is determined whether to input the caption. If the caption input is required, the caption is manually inputted. If the caption input is not required, the caption is loaded. Determine if word breaks exist during the process.

In the above step, if there is a word break, the word break is repaired through caption editing, and if there is no word break, it is determined whether a caption with an incorrect display screen is present.

As a result of performing the above step, if the caption of the display screen is incorrect, the caption is corrected by editing the caption. If there is no problem, the caption unit is generated in the screen separator unit, the caption unit list is displayed, and the frame data is loaded.

If there is no loading of the frame data in the frame data loading determination step, the frame data is loaded after selecting the caption unit to be synchronized after loading the frame data.

After synchronizing frame numbers are designated in caption units in accordance with the contents of the reproduction data obtained here, it is determined whether or not frame numbers are corrected in caption units.

If the frame number of the caption unit needs to be corrected in the step of determining whether the frame number is corrected in the above step, select the caption unit to modify the frame number, move the playback point to the position to be corrected in the frame data, and play the frame data The synchronization frame number is assigned to the caption unit in accordance with, and again the frame number of the caption unit is determined.

As a result of determining whether the frame number is corrected among the caption units, if no modification is required, it is determined that the designation of all frame numbers is completed, the caption data is stored as it is, and the caption input process ends.

The caption input method as described above has a problem in that when the display capability of the caption output display window is changed, in particular, even if the caption screen display window is increased, the caption character information is not output on the entire display screen but only the pre-programmed caption character is displayed. do.

An object of the present invention is to find the information of the playback point through the process of extracting and comparing the waveform of the key word using the first word or the first syllable of the caption unit of the caption data as a key word, and mapping the digital audio data by caption unit to the digital audio data. The present invention provides a digital audio data caption mapping method that enables the editing and retrieval of various caption information, optimizes the caption display amount according to the screen display capability of the liquid crystal display device, and simplifies the caption designation algorithm.

1 is a flowchart illustrating a conventional method of inputting digital audio caption data.

2 is a flowchart illustrating a caption unit frame data loading process of the digital audio data caption mapping process of the present invention.

3 is a flowchart illustrating a process of loading word frame data in the digital audio data caption mapping process according to the present invention.

4 is a flowchart illustrating a syllable unit frame data loading process in the digital audio data caption mapping process according to the present invention.

FIG. 5 is a detailed flowchart of the caption unit frame number automatic designation subroutine of FIG.

6 is a detailed flowchart of the word unit frame number automatic designation subroutine of FIG.

7 is a detailed flowchart of the syllable unit frame number automatic designation subroutine of FIG.

8A to 8D are diagrams illustrating a process of extracting and comparing key word waveforms in order to find playback time information by caption unit first word or first syllable of caption data and to record each caption unit.

According to another aspect of the present invention, there is provided a method of determining a caption display mode of digital audio data, determining whether the caption display screen is broken, whether a word is cut off, or whether a display screen division error occurs. Generating a caption unit and displaying a list in the order of syllables, words, lines, paragraphs, and pages after performing the step; Selecting a caption unit to be synchronized through the frame data mapping process and reproducing the frame data to determine whether the frame number is automatically designated; Performing caption unit frame number automatic designation subroutine if automatic designation in this step; if not, designating a synchronous frame number in caption unit according to the playback content if not automatic designation, and then determining whether to modify frame number among caption units; Assigning a frame number to each syllable unit and word unit by dividing a syllable unit and a word unit when it is not necessary to modify the result of performing the step; Characterized in that it comprises a.

Hereinafter, the present invention will be described with reference to the accompanying drawings.

2 is a flowchart illustrating a caption unit frame data loading process of the digital audio data caption mapping process of the present invention. As referred to herein, when the caption data input mode is selected, the first step determines whether the size of the caption display screen is determined for caption data input.

If the size of the caption display screen is not determined, the caption display screen size is determined by determining the caption output screen size.

If the caption mapping is determined in the above step, if the caption is not mapped, it is determined whether to input the caption, and if the caption input is required, the caption is manually mapped and if the caption input is not required, the caption is loaded. Determines whether word breaks exist during the process.

In the above step, if there is a word break, the word break is repaired through caption editing, and if there is no word break, it is determined whether a display caption exists.

As a result of performing the above step, if the caption of the display screen is incorrect, the caption is corrected by editing the caption. If there is no problem, the caption unit is generated in the screen separator unit, the caption unit list is displayed, and the frame data is loaded.

In this case, the caption unit is generated in the order of syllable classification information, word classification information, line classification information, paragraph classification information, and page classification information.

If there is no loading of the frame data in the frame data mapping determination step, the frame data is loaded after selecting the caption unit to be synchronized after loading the frame data.

The frame data is a multimedia file in which data such as audio / video is composed of a bit string in units of frames and includes MPEG video, MP3 audio, and the like.

When reproducing the frame data, it is determined whether the frame number is automatically designated first, and if the automatic designation mode, the caption unit frame number automatic designation subroutine is performed. If there is no need to modify, perform the syllable unit judgment step.

As the algorithm of the caption unit frame number automatic designation western routine is referred to in FIG. 5, first, it is determined whether a language is designated.

At this time, if there is no language designation, the first word among the caption data of the designated caption unit is automatically generated as TTS (Text-To-Speech) voice information, and then the generated speech data is normalized for key word speech recognition. After automatically generating TTS voice information as the first two letters of the first word among the caption data of, the generated voice data is normalized for key word speech recognition.

Following execution of the normalization process, it is determined whether a section for extracting the location of the designated caption data in the frame data is selected by the user.

In this case, if there is no selection by the user, the position that matches the normalized voice data within the frame data period is extracted through key word recognition, and the extracted position value is designated as the frame number in caption units. The automatic matching of the caption unit frame number is completed by extracting the position corresponding to the normalized speech data within the frame data section through key word speech recognition and designating the extracted position value as the caption unit frame number.

On the other hand, in the frame number automatic designation determination step, if it is not automatic designation, the synchronous frame number is assigned to the caption unit in accordance with the playback contents, and then the frame number of the caption unit is determined again. If correction is required, select the caption unit to modify the frame number, move the playback point to the position to be modified in the frame data, play the frame data, specify the sync frame number in the caption unit according to the playback contents, and then select the frame number among the caption units. Perform the step of determining again whether or not to correct.

If it is determined that the frame number is not corrected among the caption units, it is determined whether the syllable unit is corrected, and optionally, any one of the flowcharts of FIG. 3 or 4 is executed.

3 is a flowchart illustrating a case where the final syllable unit determination result of FIG. 2 is word selection.

As referred to herein, after the caption unit designation for word frame number designation, the movable view section is fixed so that the playback point can be moved only within the frame data section of the designated caption unit.

After that, a line unit to be synchronized is selected within the caption unit, the play time is moved to the frame data position to be modified within the selected line unit, and the frame number is automatically determined through the frame data reproduction process.

When reproducing the frame data, it is determined whether the frame number is automatically designated first, and if the automatic designation mode is performed, the automatic frame number sub-routine is performed. Then, the frame number is determined among the word units. If there is no need to modify it, it is determined whether there is a line unit that needs to be modified.

The algorithm for automatically specifying the syllable unit frame number western routine is first to determine the number of words in units of lines as shown in FIG. 6.

Next, after automatically generating the first word among the caption data in the designated line unit as TTS voice information, the generated speech data is normalized for key word speech recognition, and the position coinciding with the normalized voice data in the frame section of the designated line unit Is extracted through key word speech recognition.

After that, the extracted position value is designated as the frame number in the word unit, and it is determined whether all the word frame number in the designated line unit is completed.

If the result of determining whether the word frame frame number is specified is not completed, the next word among the caption data of the designated line unit is automatically generated as TTS voice data, and the previous process of normalizing the word is performed. Is completed, the word frame frame number automatic designation subroutine ends.

On the other hand, if the automatic frame number designation step is not automatic designation, the synchronization frame number is assigned to the syllable unit in the line unit according to the playback contents, and then the frame number is determined among the syllable units.

At this time, if correction is necessary, select the word unit to modify the frame number, move the playback point to the position to be corrected in the frame data, play the frame data, specify the sync frame number in the word unit according to the playback contents, and then select the frame number among the caption units. The process of re-determining whether the correction is corrected is performed again.

However, if it is not necessary to modify the frame number among the word units, it is determined whether there is a line unit that needs to be corrected in the word frame unit designation within the caption unit. If there is no line unit to be corrected, it is determined whether there is a caption unit whose word frame number designation is not completed among the caption units.

If there is a caption unit in which the above specification is not completed, the playback point moving section is fixed and returned to the beginning only within the designated caption unit. However, if the existence of the incomplete caption unit does not occur, the caption in which all the frame numbers are completed is saved and terminated.

4 is a flowchart illustrating a case in which the final syllable unit determination result of FIG. 2 is syllable selection.

As referred to herein, the movable view point is fixed so that the playback point can be moved only within the frame data section of the designated caption unit following the caption unit designation for syllable frame number designation.

After that, a line unit to be synchronized is selected within the caption unit, the play time is moved to the frame data position to be modified within the selected line unit, and the frame number is automatically determined through the frame data reproduction process.

When reproducing the frame data, it is determined whether the frame number is automatically designated first, and if the automatic designation mode is performed, the syllable unit frame number automatic subroutine is performed, and then the frame number is modified among syllable units. If there is no need to modify it, it is determined whether there is a line unit that needs correction of syllable unit frame number designation among caption units.

As shown in FIG. 7, the algorithm of the syllable unit frame number automatic designation western routine determines the number of syllables in units of lines.

Next, after automatically generating the first syllable among caption data in the designated line unit as TTS voice information, normalize the generated speech data for key word speech recognition and match the normalized voice data within the frame section of the designated line unit. Is extracted through key word speech recognition.

After that, the extracted position value is designated as the syllable unit frame number, and it is determined whether all syllable unit frame numbers in the designated line unit are completed.

If the result of determining whether the syllable unit frame number designation is completed is not completed, the next syllable of the caption data of the designated line unit is automatically generated as TTS voice data, and then the previous process of normalizing the syllable unit frame number is designated. Is completed, the syllable unit frame number automatic designation subroutine ends.

On the other hand, if the automatic frame number designation step is not automatic designation, the synchronization frame number is assigned to the syllable unit in the line unit according to the playback contents, and then the frame number is determined among the syllable units.

If correction is needed, select the syllable unit to modify the frame number, move the playback point to the position to be modified in the frame data, play the frame data, specify the sync frame number in the syllable unit according to the playback contents, and then select the frame number among the caption units. The process of re-determining whether the correction is corrected is performed again.

However, if it is not necessary to modify the frame number among syllable units, it is determined whether there is a line unit that needs to be corrected in specifying the syllable unit frame number within the caption unit. If there is no line unit to be corrected, it is determined whether there is a caption unit whose syllable unit frame number designation is not completed among the caption units.

If there is a caption unit in which the above specification is not completed, the playback point moving section is fixed and returned to the beginning only within the designated caption unit. However, if the existence of the incomplete caption unit does not occur, the caption in which all the frame numbers are completed is saved and terminated.

8A to 8D are diagrams for explaining a process of extracting and comparing key word waveforms in order to find playback time information by caption unit first word or first syllable of caption data and to record the caption time information by caption unit, and FIG. 8A is a synchronization time point. The first word or first two syllables of the selected caption data to designate the information are extracted into a waveform using TTS to form one short digital audio data. This indicates that the data is a dynamic keyword.

FIG. 8B shows a state in which the caption content of the selected caption unit is expected to be reproduced in the digital audio data. In this way, it is shown that the expected subtitle reproduction interval is defined as the recognition interval by extracting the unique waveform value through the voice recognition of the caption data.

FIG. 8C illustrates a process of searching for a point that is determined to be similar to the previously generated dynamic keyword within the recognition interval by comparing the output signal waveform of the dynamic keyword with the signal waveform within the recognition interval. At this time, in order to accurately compare the signal waveforms, the data between the dynamic keywords and the recognition parameters are normalized according to a formula.

FIG. 8D illustrates a process of storing location information of a point where a dynamic key word and a signal are determined to be the same (or similar) within a recognition period, in a caption unit.

As described above, when mapping caption data to digital audio data, the playback point information can be quickly obtained by extracting and comparing the waveform of the key word using the first word or the first syllable as the key word of the caption unit of the caption data. By capturing and mapping the digital audio data by caption unit, it is possible to edit and import various caption information based on digital audio data, and to optimize the caption display amount according to the screen display capability of the liquid crystal display device. It has a unique effect that can simplify the designation algorithm.

Claims (6)

Determining the size of the caption display screen, whether the word is cut off during the caption display, and whether there is a display screen division error when the caption data input mode is selected on the digital audio data; Generating a caption unit and displaying a list in the order of syllables, words, lines, paragraphs, and pages after performing the step; Selecting a caption unit to be synchronized through the frame data loading process and reproducing the frame data to determine whether the frame number is automatically designated; Performing caption unit frame number automatic designation subroutine if automatic designation in this step; if not, designating a synchronous frame number in caption unit according to the playback content if not automatic designation, and then determining whether to modify frame number among caption units; And if it is not necessary to modify the result of performing the step, assigning a frame number to each syllable unit and word unit by dividing the syllable unit and the word unit. The method of claim 1, The caption unit frame number automatic designation subroutine includes determining whether to designate a language; In this step, if there is no language designation, the first sentence of the caption data of the designated caption unit is automatically generated as TTS voice information, and the generated voice data is normalized. If the language designation is specified, the first two letters of the first sentence of the caption data of the designated caption unit are normalized. Normalizing the generated voice data after automatically generating the TTS voice information; Determining whether a section for extracting the position of the designated caption data in the frame data has been selected by the user following execution of the normalization process; If there is no selection by the user, the location that matches the normalized voice data within the frame data is extracted through key word recognition, and the extracted position value is designated as the frame number in caption units. Extracting a position corresponding to the normalized speech data within the interval through keyword recognition and designating the extracted position value as a frame number in caption units; Digital audio data caption synchronization method comprising a. The method of claim 1, The caption frame number of word unit is fixed by moving playback point section, selects line unit to synchronize within caption unit, moves playback point to frame data position to be modified within selected line unit and plays frame data automatically. Determining whether to designate; In the above step, if the automatic frame subnumber is assigned to the word frame frame number, the frame number is corrected in the word unit. If not, the sync frame number is assigned to the syllable unit in the line unit according to the playback contents. Modifying the number; It is determined whether there is a line unit that needs to be corrected in word frame frame number designation within caption unit, and if there is no line unit that needs to be corrected, it is determined whether there is a caption unit whose word frame number designation is not completed among caption units. Making a step; Storing and ending captions in which all frame number designations are completed when designation is completed in the step; Digital audio data caption synchronization method comprising a. The method of claim 3, The syllable unit frame number automatic designation western routine detects the number of words in a line unit and automatically generates the first word of caption data in the designated line unit as TTS voice information; Thereafter normalizing the generated voice data for key word speech recognition and extracting a position corresponding to the normalized voice data within the designated line-by-frame frame through key word speech recognition; Designating the extracted position value as a frame number in word units and determining whether all word frame number numbers in the designated line units are completed; Repeating the step of automatically generating the next word of the caption data in the designated line unit as the TTS voice data until the designation of the word unit frame number designation is completed; Digital audio data caption synchronization method comprising a. The method of claim 1, The caption frame number designation of syllable unit is to fix the moving playback point section, select the line unit to be synchronized within the caption unit, move the play time to the frame data position to be modified within the selected line unit, and play the frame data automatically. Determining whether to designate; In the above step, if the syllable unit frame number automatic subroutine is executed, the frame number of the syllable unit is corrected.If not, the sync frame number is assigned to the syllable unit in the line unit according to the playback contents. Modifying the number; In the syllable unit frame number designation within the caption unit, it is determined whether there is a line unit that needs to be corrected. If there is no line unit that needs to be corrected, it is determined whether there is a caption unit whose syllable unit frame number designation is not completed among the caption units. Making a step; Storing and ending captions in which all frame number designations are completed when designation is completed in the step; Digital audio data caption synchronization method comprising a. The method of claim 5, The syllable unit frame number automatic designation western routine detects the number of syllables in the line unit and automatically generates the first syllable of the caption data in the designated line unit as the TTS voice information; Thereafter normalizing the generated voice data for key word speech recognition and extracting a position corresponding to the normalized voice data within the designated line-by-frame frame through key word speech recognition; Designating the extracted position value as a frame number in syllable units and determining whether all syllable unit frame numbers in the designated line units are completed; Repeating the step of automatically generating the next syllable among the caption data in the designated line unit as TTS voice data until the determination result of the syllable unit frame number designation is completed; Digital audio data caption synchronization method comprising a.