KR20040034338A - Sync signal insertion/detection method and apparatus for synchronization between audio contents and text - Google Patents

Abstract

PURPOSE: A synchronization signal insertion and detection method and system is provided to synchronize audio contents with text by inserting a synchronization signal into an audio file so that it minimizes an effect of text synchronization to an audio quality. CONSTITUTION: The method comprises several steps. An MP3 audio file to be played is selected and is divided into frame units(S301). A frame analysis is performed for each divided frame(S303). In a case that a space for inserting a synchronization signal into is necessary(S305), an arbitrary space for a stuffing space is generated(S307). Then, one byte is newly allotted for the stuffing space and position addresses of all the posterior frames are increased by one byte(S309). It is determined whether a synchronization signal has to be inserted into a corresponding frame(S311). If so, the synchronization signal is inserted into a stuffing space(S313).

Description

SYNC SIGNAL INSERTION / DETECTION METHOD AND APPARATUS FOR SYNCHRONIZATION BETWEEN AUDIO CONTENTS AND TEXT}

The present invention relates to a method and apparatus for synchronizing between digital audio content and corresponding text in a portable digital playback device.

In recent years, in response to the development of computer technology, a technology for reproducing audio content using a computer has been rapidly developed. Accordingly, attention has been paid to a function of visually displaying the contents of the audio contents while playing the audio contents. For example, the technology of displaying the lyrics on the screen while playing the audio contents related to the song corresponds to this. .

Referring to Fig. 10, a description will be given of a configuration for simultaneously displaying content contents during reproduction of audio contents in the prior art.

First, an audio content to be played back and a text file storing the content of the audio content are provided. FIG. 10 is a diagram illustrating a reconstruction of a text file storing contents of conventional audio content in the form of a table. In Fig. 10, the text file stores not only the contents of the audio contents but also a playback time for visually displaying the contents of the audio contents. In the example of FIG. 10, while the compressed voice or music file is being reproduced, a reproduction time point indicating the time to output the text is stored in units of 1/1000 second.

For example, at a playback time of 0000040 ms, audio content is played back, and a character string corresponding to the audio content is visually output through a predetermined display. As the audio content is played back, at the playback time 0001055 ms, at the same time as the audio content is played, a string "while playing music or an audio file" is output.

That is, the playback time is monitored while the audio content is being played back, so that the output string is output when the playback time coincides with the playback time of the output string shown in the table.

The structure of such a text file is substantially similar to that of, for example, a ".smi file" for outputting subtitles in a video, and is suitable for a case where sufficient resources are available, such as a computer.

However, there is a limit to the resources available when synchronizing the digital audio content and the corresponding text in the digital portable playback apparatus by the above-described method. Therefore, it is not practical to monitor the playback time in ms of audio content in the digital portable playback device and output the text in accordance with the minute playback time. Therefore, the above-described method of outputting text based on the information of the table by storing the reproduction time and the text in the form of a table in a text file is not suitable for the digital portable playback apparatus.

In addition, in the conventional method of outputting text, since text information is arbitrarily output on the LCD screen according to the time to be reproduced, there is a problem that the content actually reproduced does not coincide with the content output to the liquid crystal.

Next, a method of embedding a synchronization signal into a digital audio content as a watermark through frequency conversion or the like will be described. In general, the watermarking technology refers to a technology for storing information of a work which is not recognized by the general public in a sound recording for copyright protection of a work and for determining whether the work is forged or forged. Since watermarking technology conceals user-defined information in the actual sound source of the copyrighted work, it is common to use robust watermarks that are robust against signal processing attacks, compression conversion, etc., and that are difficult to remove for malicious purposes. to be.

Since watermarking inserts data into a sound source of digital content, a large amount of computation and computation must be performed because a fairly complicated calculation process must be performed in order to detect hidden information again. Since watermarking technology usually consumes a considerable amount of resources to implement a DSP, it is difficult to use a portable digital playback device such as a portable MP3 player using a DSP. In addition, an additional function that consumes a lot of resources is undesirable in view of the limited battery life of the portable playback device. In particular, since most audio data is in a format for compressing target content, conventional watermarking techniques are not available.

A technique for hiding information in compressed data is disclosed in MP3Stego (Computer Laboratory, Cambridge, August, 1998) proposed by F. Petitcolas. This technique has a problem in that high-speed insertion processing is not possible because data is concealed during the compression of the sound source.

Also, Non-Invertible Watermarking Methods For MPEG Encoded Audio (Security and watermarking of Multimedia Contents, January 1999) proposed by L. Qia and K. Nahrstedt has a high risk of altering the sound source of MP3 and has a limited amount of concealable information. There is this.

In addition, high-speed extraction may be possible in A compressed-domain watermarking algorithm for MPEG Audio Layer 3 (ACM Multimedia 2001, Septemper 30-October 5, Ottawa, Ontario, Canada) proposed by DK Koukopoulos and YC Stamatiou, but high-speed insertion processing is not possible. There is a problem.

The present invention has been made to solve the above-described problems, and minimizes the effect of text synchronization on sound quality, and enables high-speed insertion / processing while matching playback time and text output time of audio content. It is an object of the present invention to provide a synchronization signal insertion method for inserting a synchronization signal into an audio file so as to synchronize text.

It is also an object of the present invention to provide a method for preventing excessive resource consumption in an audio content playback apparatus during playback of audio content and output of text synchronized with it.

Another object of the present invention is to provide a synchronization signal detecting method and apparatus for detecting a synchronization signal from an audio file into which a synchronization signal is inserted.

1 is a conceptual diagram illustrating an overall process for synchronizing an audio file and a corresponding text in a digital portable playback device.

2 is a diagram illustrating a structure of an MP3 frame.

3 is a flowchart illustrating a synchronization signal insertion process according to a first embodiment of the present invention.

4 is a flowchart illustrating a synchronization signal insertion process according to a second embodiment of the present invention.

FIG. 5 is a schematic diagram illustrating an audio file in which a synchronization signal is inserted according to a second embodiment of the present invention in units of frames.

6 is a conceptual diagram illustrating a process of synchronizing text with a voice file generated by the TTS technology.

7 is a schematic diagram schematically illustrating a synchronization signal detection process according to the present invention.

8 is an internal configuration diagram when a synchronization signal detection apparatus for text synchronization according to the present invention is implemented in a DSP of a portable digital reproduction apparatus.

9 is an internal configuration diagram when the DSP is implemented in a portable digital playback device.

FIG. 10 is a diagram illustrating a reconstruction of a text file storing contents of conventional audio content in the form of a table.

Explanation of symbols on the main parts of the drawings

101: Text 103: Audio File

105: text synchronization device 107: manager program

109: Portable Storage

201: header 203: minor information

205 main data 207 stuffing space

In order to achieve the above object, the present invention provides a first portion in which audio content is stored, a second portion including at least information about the size of the first portion, and a portion other than the first portion and the second portion. CLAIMS 1. A method for inserting a synchronization signal into an audio file comprising a plurality of frames each having a third portion, the method comprising: obtaining information about a size of the first portion of the frame from a second portion of the frame; Determining a starting position and size of a third portion of the frame based on the obtained information; And inserting at least a portion of a sync signal into the third portion of the frame.

Here, the first part includes header information of the audio file, the second part includes the audio content, and the third part is a part which is not used to play audio content of the audio file. In addition, the third part includes an area indicating whether a synchronization signal is present and an area indicating the content of the synchronization signal.

The synchronization signal may also include information regarding the position of the text corresponding to the first portion of the frame, and inserting at least a portion of the synchronization signal into the third portion of the frame comprises: Determining whether to insert a synchronization signal into the third portion; And in response to the non-insertion determination of the synchronization signal, inserting text information corresponding to the first portion of the frame into the third portion of the frame.

In the inserting of at least a part of the synchronization signal into the third portion of the frame, the synchronization signal insertion space in the third portion may be compared with the magnitude of the synchronization signal to insert the synchronization signal in the third portion. When the space is smaller than the size of the synchronization signal, it is preferable to insert a portion of the synchronization signal with the same size as the synchronization signal insertion space into the third portion.

In addition, the audio content may be generated by converting the text to text-to-speech (TTS).

Meanwhile, the present invention has a first part in which audio content is stored, a second part including information on the size of at least the first part, and a third part which is a part other than the first part and the second part, respectively. A method of detecting a synchronization signal from an audio file including a plurality of frames, the method comprising: extracting information about a start position and a size of the third portion based on the information about the size of the first portion; Analyzing the third portion to determine whether a synchronization signal is present; And in response to determining the presence of the synchronization signal, obtaining at least a portion of the synchronization signal from the third portion.

Here, the first part includes header information of the audio file, the second part includes the audio content, and the third part is a part which is not used to play audio content of the audio file. In addition, the third part includes an area indicating whether a synchronization signal is present and an area indicating the content of the synchronization signal.

The method may further include extracting text information from the third portion in response to the determination of the absence of the synchronization signal. After analyzing the contents of the synchronization signal, the location of the corresponding text may be determined based on the analysis. The method may further include selecting.

The method may further include combining at least a portion of the synchronization signal with at least a portion of the synchronization signal of a subsequent frame when at least a portion of the synchronization signal obtained from the third portion is not the same as the synchronization signal. .

Meanwhile, the present invention has a first part in which audio content is stored, a second part including information on the size of at least the first part, and a third part which is a part other than the first part and the second part, respectively. In the apparatus for detecting a synchronization signal from an audio file including a plurality of frames, based on the information about the size of the first portion, extracting information about the start position and size of the third portion, and the third portion Analyzing the signal to determine whether there is a synchronization signal; And a synchronizing signal acquiring unit for acquiring at least a portion of the synchronizing signal from the third portion in response to the existence determination of the synchronizing signal.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram illustrating an overall process for synchronizing an audio file and a corresponding text in a digital portable playback device.

Referring to FIG. 1, first, an audio file 103 and text 101 corresponding thereto are input to the text synchronization device 105. Using the inputted information, the text synchronizing apparatus 105 directly receives a time point at which each lyrics should be output from the user. The information received from the user may be composed of information to which the text to be output and the playing time are connected, respectively. The text synchronizing apparatus 105 inserts information indicating the position of text for text output corresponding to the predetermined position of the audio file 103 according to the synchronization signal insertion method according to the present invention. The manager program 107 receives the synchronized MP3 file and the text from the text synchronizing apparatus 105, and downloads them to the portable playback apparatus 109.

Then, in the case of reproducing the audio file 103 in the portable playback device 109, if a synchronization signal is detected during the reproduction of the audio file, the synchronization signal is analyzed to find the position of the text indicated by the synchronization signal, The character string to be output through the display means of the portable playback device 109.

Hereinafter, although the format of the music file is described as MP3 in the embodiment of the present invention, in the case of music files stored according to other audio file formats such as WMA, AAC, and AC3, the synchronization signal insertion method according to the present invention may be applied or applied. It will be apparent to those skilled in the art that this can be accomplished.

2 is a diagram illustrating a structure of an MP3 frame. Referring to FIG. 2, the structure of an MP3 frame includes an MP3 audio file composed of a plurality of consecutive frames, each frame comprising a header 201, side information 203, and main information having 12 bits of sync bits. Data 205, and stuffing space 207.

The header 201 and the sub information 203 store general information on the structure of the frame and the like, including sync. In the main data 205, audio content is losslessly compressed and stored according to a Huffman Coding scheme. The lossless compressed main data 205 is stored in units of bytes, and as a result of Huffman coding, redundant bits are generated in which the content of the audio content is not included at all. Such a surplus bit is called a stuffing bit, and a part of the stuffing bit is called a stuffing space. In other words, these bits are empty space that is not used at all when playing music. Since the stuffing space 207 is a bit for making the size of the frame including the main data 205 in units of bytes, the size of the stuffing space 207 is equal to the size of the main data 205 generated by Huffman coding audio content. Is determined accordingly.

As will be described in more detail below, the present invention inserts a synchronization signal into the stuffing space by using the structural characteristics of the frame.

3 is a flowchart illustrating a synchronization signal insertion process according to a first embodiment of the present invention. Referring to FIG. 3, first, when an MP3 audio file to be reproduced is selected, the MP3 audio file is divided into frames (S301).

For each divided frame, frame analysis is performed (S303). The frame analysis analyzes the header 201 and the sub information 203 to obtain information about the start position and the size of the main data 205. Then, based on the information about the size of the main data 205, the size and position of the stuffing space 207 is obtained.

Depending on the size of the main data 205, it may be determined that the stuffing space 207 does not exist. Also in this case, when it is determined that a space for inserting the synchronization signal is necessary (S305), a space for the stuffing space 207 may be arbitrarily generated (S307). At this time, one byte is newly allocated for the stuffing space, and thus, all subsequent frames are reconstructed so as to push backward by one byte (S309).

Then, it is determined whether or not a synchronization signal should be inserted into the frame (S311). Whether to insert the synchronization signal may be determined according to information previously input from the user. For example, the user may directly input which part of the text should be output at some time while playing the audio file through a predetermined input device of the text synchronization device. It may also be automatically determined as in the case of the TTS method described later. When the synchronization signal is to be inserted, the synchronization signal is inserted into the stuffing space (S313). Since the size of the synchronization signal is generally larger than the number of bits of the stuffing space, not all of the one synchronization signal is inserted into one stuffing space, but at least a part of the synchronization signal is inserted into one stuffing space. One sync signal may be inserted into a plurality of stuffing spaces. In an exemplary embodiment, the stuffing space includes a portion indicating the presence of the synchronization signal, and a portion indicating the position of the text and the number of characters of the output text as the content of the synchronization signal. How many bits of the synchronization signal are inserted into the corresponding frame depends on how many bits are given in the stuffing space.

By repeating the above process for each frame, a synchronization signal is inserted into an audio file consisting of frames.

Therefore, through the above-described configuration, by providing a synchronization signal for inserting a synchronization signal into the audio file so as to synchronize the audio content and the text, the audio content reproduction apparatus at the time of reproduction of the audio content and the output of the text synchronized therewith. Excessive resource consumption may not occur in the system.

Next, a second embodiment of the present invention will be described with reference to FIGS. 4 and 5. 4 is a flowchart illustrating a synchronization signal insertion process according to a second embodiment of the present invention.

Although not shown in FIG. 4, steps S301 to S309 of FIG. 3 exist the same as before step S411 of FIG. 4, but will be omitted for convenience of illustration and description.

First, it is determined whether the synchronization signal needs to be inserted (S411).

If the synchronization signal does not need to be inserted, text is inserted into the stuffing space (S415). Since the length of the text string is generally larger than the number of bits of the stuffing space, it inserts at least a portion of the text string into one stuffing space, rather than inserting all of the given text strings into one stuffing space. That is, one text string is inserted into a plurality of stuffing spaces.

FIG. 5 is a schematic diagram illustrating an audio file in which a synchronization signal is inserted according to a second embodiment of the present invention in units of frames. In FIG. 5, the audio file is schematically illustrated in frames. For each frame, the frame corresponding to the text information insertion includes text information, and the frame corresponding to the text output time point includes a synchronization signal. No information may be inserted into the stepping space even in the frame corresponding to the text information insertion, which means the standby area as described above. The text information to be output is first inserted into one or more frames so that the playback time of the frame including the synchronization signal is the point of time of outputting the text inserted in the frame before it. After inserting all the text information to be output, it is in the waiting state until the synchronization signal is inserted. In the standby state, all stuffing bits present in each frame are initialized to '0' without inserting additional information into the frame. Then, when the position of the current frame coincides with the time information to output the text, a synchronization signal is inserted.

4, when the synchronization signal is to be inserted, the synchronization signal is inserted into the stuffing space (S413). As described above with reference to FIG. 3, since the size of the synchronization signal is generally larger than the number of bits of the stuffing space, all of one synchronization signal may be inserted into one stuffing space. It can also be inserted into the stuffing space. That is, one synchronization signal may be inserted into the plurality of stuffing spaces. It is sufficient that the synchronization signal inserted into the stuffing space includes only a portion indicating the presence of the synchronization signal. In the reproduction of the audio file, since the information stored in the stuffing space of the previous frames of the frame in which the synchronization signal is detected is a piece of text information, combining these results in the text to be output to the display upon detection of the presence of the synchronization signal. to be.

By repeating the above process for each frame, text corresponding to the synchronization signal and the audio content is inserted into the audio file consisting of the frames.

Meanwhile, the process of synchronizing the audio file and the lyrics text according to the present invention may be generated using a text-to-speech (TTS) engine. 6 is a conceptual diagram illustrating a process of synchronizing text with a voice file generated by the TTS technology.

TTS is a technology for speech synthesis of text into a voice file. In converting text characters into audio files, the TTS engine 603 builds a phoneme DB in minimum pronunciation units for each language of the country, and then A voice signal is generated by synthesizing the retrieved phoneme DB considering the context of front and back. In the configuration of the present invention described above with reference to FIG. 1, the position of text for synchronizing with an audio file must be directly input from a user. However, in the case of speech synthesis by TTS, a text file corresponding to the voice file is generated at the same time. Because the location of the text is automatically detected, no separate user input process is necessary.

Hereinafter, a synchronization signal detection process according to the present invention will be described.

7 is a schematic diagram schematically illustrating a synchronization signal detection process according to the present invention.

MP3 audio files are stored in memory. In response to the playback command for the MP3 audio file, information of the MP3 audio file is read from the memory (S701). The read MP3 audio file is provided for frame analysis in the form of an MP3 stream.

Thereafter, the audio file transmitted in the form of an MP3 stream is divided in units of frames (S703). Each frame can be determined from the position of the previous frame and the header and sub information of each frame.

Then, the size of the audio content is extracted using the header and sub information for each frame. Since the bit size and position of the stuffing space can be known based on the size of the audio content, it is possible to grasp the bit size and position of the stuffing space accordingly. That is, information about the stuffing space is identified (S705). Thereafter, information about the presence of the stuffing space and the position and size (if any) is provided for the synchronization signal and text construction.

Thereafter, the content of the detected sync signal is analyzed to form a sync signal and a text (S707). In the case of the first embodiment, the position of the text in the text file indicated by the synchronization signal and the length of the character string to be displayed are determined, and the character string portion is read from the text file. On the other hand, in the case of the second embodiment in which the text is included in the MP3 audio file, when there is no synchronization signal, the bit content of the stuffing space is read out and stored in a separate memory space continuously, and the synchronization signal When the presence of is detected, the content stored in the memory space is output as text. After output as text, the contents are removed from the memory space. Then, the text string is provided for output to the LCD.

Thereafter, the LCD controller (not shown) controls the LCD to erase the string currently output on the LCD and output a new string (S709). In this case, if it is necessary to output text longer than a string that can be printed on the LCD at the same time, the character string can be automatically scrolled from right to left, and this scrolling process is known to those skilled in the art.

The sync signal detecting apparatus of FIG. 7 may be implemented in a digital portable playback apparatus as shown in FIGS. 8 and 9. Although it is common to implement in the DSP, the text synchronization operation is controlled by all external devices in MICOM, so if there are enough resources in MICOM, it is advantageous to implement in MICOM as shown in FIG. Since the processing speed and memory required to implement synchronization by the method proposed by the present invention are very small, it is possible to process in MICOM.

8 is an internal configuration diagram when the synchronization signal detection apparatus for text synchronization is implemented in the DSP of the portable digital reproduction apparatus, and FIG. 9 is an internal configuration diagram when the DSP is implemented in the portable digital reproduction apparatus. .

8 and 9 are internal configuration diagrams of a general playback apparatus. When a user presses a play button, the microcomputer brings a file name to be played back. After getting the name of the file to play, the data of the file is read and transferred to the buffer, and the DSP decodes the compressed data in the buffer to play music through the speaker.

In this process, when the present invention which displays the voice information of the lyrics or the file to be reproduced on the liquid crystal is inserted, the overall structure is changed as follows. The process of importing a file to be played in Micom is the same. After retrieving the file to be reproduced, the data read from the file is transferred to the buffer, and the synchronization signal detector determines whether the transferred data has a synchronization signal. In this case, when the synchronization signal is found by the synchronization signal detector, the synchronization signal is found by the controller of the microcomputer, and the content of the found synchronization signal is informed. The microcomputer's LCD controller sends the information from the sync signal detector to the liquid crystal display.

The difference between FIG. 8 and FIG. 9 differs only in where the sync signal detector is located. The overall execution procedure is the same regardless of the shape of the portable reproducing apparatus.

The invention has been described with reference to specific embodiments for specific applications. Those of ordinary skill in the art, having access to the present teachings, may know additional variations, applications, and embodiments within the scope.

Accordingly, the appended claims are intended to cover any and all such applications, modifications, and embodiments within the spirit of the invention.

The present invention provides a function capable of displaying lyrics or voice content of music automatically reproduced while reproducing a music file or an audio file by adding a text synchronization device to the digital portable player.

The present invention detects a synchronization signal hidden in a music file in real time while the compressed file is being reproduced, and displays it on the LCD screen in synchronization with the current playback time of the content file. Therefore, the user can check the content currently being played back through the LCD screen of the playback device. In addition, by hiding all the information in the digital content until the text information and the text should be output, the user does not need to additionally store the text file or other information.

In particular, the present invention can be used effectively in a digital portable playback device for language learning because the present invention can comprehensively utilize textbooks for foreign language learning as well as lyrics of general music.

Claims (14)

A plurality of frames each having a first portion in which audio content is stored, a second portion containing information about the size of the first portion, and a third portion that is a portion other than the first portion and the second portion; In a method of inserting a synchronization signal into an audio file, Obtaining information about the size of the first portion of the frame from the second portion of the frame; Determining a starting position and size of a third portion of the frame based on the obtained information; And And inserting at least a portion of the synchronization signal into the third portion of the frame. The method of claim 1, The first portion includes header information of the audio file, The second portion includes the audio content, And the third portion is a portion not used for reproduction of audio contents of the audio file. The method of claim 1, And the third portion includes an area indicating whether a synchronization signal is present and an area indicating the content of the synchronization signal. The method of claim 1, And the synchronization signal includes information about a position of text corresponding to the first portion of the frame. The method of claim 1, Inserting at least a portion of a sync signal into the third portion of the frame, Determining whether to insert a synchronization signal into the third portion of the frame; And And in response to the non-insertion determination of the synchronization signal, inserting text information corresponding to the first portion of the frame into the third portion of the frame. The method according to any one of claims 1 to 5, Inserting at least a portion of a sync signal into the third portion of the frame, Comparing the size of the synchronization signal insertion space and the synchronization signal in the third portion, when the synchronization signal insertion space in the third portion is smaller than the size of the synchronization signal, the same size as the synchronization signal insertion space And inserting a portion of the synchronization signal into the third portion. The method of claim 1, And the audio content is generated by text-to-speech (TTS) conversion of the text. A plurality of frames each having a first portion in which audio content is stored, a second portion containing information about the size of the first portion, and a third portion that is a portion other than the first portion and the second portion; In the method for detecting a synchronization signal from an audio file, Extracting information about a start position and a size of the third part based on the information about the size of the first part; Analyzing the third portion to determine whether a synchronization signal is present; And In response to determining the presence of the synchronization signal, obtaining at least a portion of the synchronization signal from the third portion. The method of claim 8, The first portion includes header information of the audio file, The second portion includes the audio content, And the third portion is a portion not used for reproduction of audio contents of the audio file. The method of claim 8, And the third portion includes an area indicating whether a synchronization signal is present and an area indicating the content of the synchronization signal. The method of claim 8, And in response to determining the absence of a synchronization signal, extracting text information from the third portion. The method of claim 8, And analyzing the content of the synchronization signal, and then selecting a position of a corresponding text based on the analysis. The method according to claim 8 to 12, And combining at least a portion of the synchronization signal with at least a portion of the synchronization signal of a subsequent frame if at least a portion of the synchronization signal obtained from the third portion is not the same as the synchronization signal. Signal detection method. A plurality of frames each having a first portion in which audio content is stored, a second portion containing information about the size of the first portion, and a third portion that is a portion other than the first portion and the second portion; In an apparatus for detecting a synchronization signal from an audio file, On the basis of the information on the size of the first portion, information on the start position and size of the third portion is extracted, and the third portion is analyzed to determine whether there is a synchronization signal presence plate to determine the presence of a synchronization signal. government; And And a synchronizing signal acquiring unit for acquiring at least a portion of the synchronizing signal from the third portion in response to the determination of the existence of the synchronizing signal.