KR101379697B1 - Apparatus and methods for synchronized E-Book with audio data - Google Patents

Abstract

The e-book authoring apparatus according to the present invention receives voice data and document data, converts the information of the voice data into a text in a sentence unit, and converts the time data of the voice data into a time code. A sentence extractor which stores the first sentence pool together, extracts text from the document data in sentence units, and stores the text as a second sentence pool. The sentence of the second sentence pool is based on a preset similarity. A sentence matching unit for determining a sentence (first sentence) of the first sentence pool matched with a sentence), and synchronizing and storing time information of the first sentence after synchronizing with the second sentence The sentence, every one of the first sentence pool, the second sentence pool and the memory unit for storing the synchronization information, and the first sentence pool or the second sentence pool, the sentence every until there is no sentence to match Unit and a control unit for controlling the matching and synchronization via the synchronization unit.
According to the present invention, it is possible to produce an e-book in which voice data and document data are synchronized in units of sentences, and by synchronizing the position information and voice data of sentences in the document data, even if a random sentence is selected from the document data, the synchronized voice data is immediately available. Can be accessed.

Description

Apparatus and methods for synchronized E-Book with audio data}

The present invention relates to an e-book authoring apparatus and method synchronized with audio data (voice data). More specifically, the present invention relates to an electronic book authoring apparatus and a method in which voice data having the same content as the text of the electronic book is synchronized with the text.

E-books (E-books) are digitalized content that can be viewed in the form of books or magazines using digital devices such as computers, e-book readers, and smart phones.

BACKGROUND ART Conventional e-books generally display text-type documents on the display screen of the e-book device, and a user can use the text to view the text through the display.

In addition, there is an audio book content method similar to an e-book, which converts a text-type document into TTS (Text to speech) or stores a voice that reads the text in a digital file format. Say the way.

In the fields of e-learning, not only text or audio e-books, but also text-based e-books and voice data about the contents as shown in FIG. There is this.

For example, a young child learning a letter or a student studying a foreign language can learn efficiently by simultaneously listening to and viewing data in a text form and a voice form about the same content. In addition, if an audiobook of the same content for any sentence is immediately accessible at the same time in the middle of the text-type e-book, the learning efficiency can be expected to be improved.

However, conventional e-books and audiobooks have been developed separately from each other, and even in the case of integrating them, text and video as in Korean Patent Publication No. 10-2009-0060040 (“Interactive digital audiobook production system and its operation method”). There is a problem in that this unit is not synchronized in units of sentences, but only in chapters or pages.

Domestic Publication No. 2009-0060040 (2009.06.11) Title of the Invention: Interactive Digital Audiobook Production System and Its Operation Method

An object of the present invention is to provide an e-book authoring apparatus and method for producing an e-book in which voice data and document data are synchronized in units of sentences.

Another object of the present invention is to provide an e-book authoring apparatus and method for synchronizing positional information of a sentence and text data in document data so that the synchronized voice data can be directly accessed even when an arbitrary sentence is selected from the document data.

According to an e-book authoring method of an e-book authoring apparatus, (a) receiving voice data and document data, (b) text-recognized text by speech recognition of the voice data; And converting the voice data into a first sentence pool along with time information of the voice data, extracting text from the document data in units of sentences, and storing the second sentence pool as a second sentence pool, (c (C) determining a sentence (first sentence) of the first sentence pool that matches the sentence (second sentence) of the second sentence pool based on a preset similarity level, (d) the first sentence Synchronizing and storing time information of the time code with the second sentence and (e) in any one of the first sentence pool and the second sentence pool, until there is no sentence to match; And repeating step (d). The.

In addition, in the e-book authoring method, when text is extracted from the document data in units of sentences in step (b), sentence position information of the extracted sentence with respect to the document data may also be extracted and stored as a second sentence pool. .

In addition, in the e-book authoring method, after the step (c), when the length of the first sentence and the second sentence is different from (c-2), the second sentence is adjusted based on the first sentence. It may further comprise a step.

In addition, in the e-book authoring method, in the step (d), the synchronization may synchronize the time information with the sentence position information of the second sentence.

To solve the problem of the present invention, an e-book authoring apparatus receives voice data and document data, and recognizes the voice data information, converts the voice data into text in a sentence unit, and then converts time information of the voice data into time. A sentence extractor that stores the first sentence pool along with a code, extracts text from the document data in a sentence unit, and stores the text as a second sentence pool, based on a preset similarity, based on a preset similarity. A sentence matching unit for determining a sentence (first sentence) of the first sentence pool matching the (second sentence), and synchronizing a time code of the first sentence with the second sentence In any one of a synchronization unit for storing, the first sentence pool, the second sentence pool, and a memory unit for storing synchronization information, and the first sentence pool or the second sentence pool, until there are no sentences to match. Group comprises a sentence matching unit and a control unit for controlling the matching and synchronization via the synchronization unit.

According to the present invention, an electronic book in which voice data and document data are synchronized in sentence units can be produced.

In addition, according to the present invention, by synchronizing the positional information of the sentence and the speech data in the document data, the synchronized speech data can be directly accessed even when an arbitrary sentence is selected from the document data.

1 is a block diagram of an e-book storage device.
2 is a flowchart of an e-book authoring method of an e-book storage device.
3 is a diagram illustrating the concept of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

The embodiments disclosed herein should not be construed or interpreted as limiting the scope of the present invention. It will be apparent to those of ordinary skill in the art that the description including the embodiments of the present specification has various applications. Accordingly, any embodiment described in the Detailed Description of the Invention is illustrative for a better understanding of the invention and is not intended to limit the scope of the invention to embodiments.

The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the following detailed description. Also, although one or more functional blocks of the present invention are represented as discrete blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

1 is a block diagram of an e-book authoring method of an e-book authoring apparatus according to the present invention.

According to FIG. 1, an e-book authoring apparatus according to the present invention receives voice data and document data, processes the voice data and the document data, and stores the voice data and the document data together in a first sentence pool and a second sentence pool, respectively. The sentence extracting unit 110, a sentence matching unit 120 for determining a sentence (first sentence) of the first sentence pool that matches the sentence (second sentence) of the second sentence pool, and the first sentence. A synchronization unit 130 for synchronizing and storing time information of a sentence after synchronizing with the second sentence, and matching through the sentence matching unit 120 and the synchronization unit 130 until there is no sentence to match. It includes a control unit 140 for controlling the synchronization.

The sentence extracting unit 110 receives the voice data and the document data, and recognizes the information of the voice data with respect to the input data, converts the information into text in a sentence unit, and then time information of the voice data. Code is stored in the first sentence pool, and text is extracted from the document data in sentence units and stored in the second sentence pool.

In order not to obscure the gist of the present invention, a detailed method and a conventional method of converting voice data into text through voice recognition technology will be omitted.

The voice data received from the sentence extractor 110 refers to voice data read by a person or a text to speech (TTS) device, and the document data refers to data in the form of a digital file including the specific text. do.

The sentence extracting unit 110 converts the voice data into text in units of sentences. That is, the speech data is grammatically converted into text in an appropriate sentence unit.

In this case, the entire voice data may be converted into text and then divided into sentence units, or the text data of the voice data and sentence division may be simultaneously performed.

Speech data may be converted into units of sentences through semantic analysis and grammatical analysis of text sentences, or sentence units of voice data may be converted by simultaneously analyzing a blank length of a voice data and sentence meaning analysis.

In the case of using a blank length of voice data, the following method may be used. After the voice data is filtered through noise elimination filtering and a signal below a predetermined threshold, the entire voice data scan is used to check the lengths of the blank portions without the voice signal. The length distribution of the spaces may be examined to determine the length of spaces between the words, and the left and right data may be divided into separate sentences based on a portion of the space in which the space exceeding the space exists in the input voice data.

In addition, the sentence unit division may be adjusted by analyzing the syntax and semantics of the sentence together with the blank length of the voice data.

The sentence extracting unit 110 stores the data converted into text in sentence units as a first sentence pool along with time information on the voice data.

The time information refers to information on a time at which voice data corresponding to the sentence conversion data is located in the entire voice data.

For example, in the case of converting speech data having a length of 10 minutes into sentence text, a case in which a specific A sentence exists 4 minutes from the start of the speech data can be considered. In this case, the time information may be expressed in an absolute manner of “00:04:00” or “240/600 (sec)” in a manner relative to the entire voice data.

The sentence extractor 110 extracts text from the input document data in sentence units and stores the text in a second sentence pool.

The document data may be a document file such as a commonly used MS-Word file or an Adobe PDF file, a web document such as HTML, or an existing e-book. In addition, it is also possible to digitally input digital streams, including text, and to receive an Internet link in the form of an address of an Internet web document.

When extracting document data as text in sentence units, it is possible to use semantic analysis and grammatical analysis of text sentences, or to simultaneously use punctuation marks present in document data.

In addition, when extracting text from the document data in sentence units, the sentence extractor 110 may also extract sentence position information of the extracted sentence with respect to the document data and store it as a second sentence pool. That is, up, down, left, and right coordinates of the corresponding page of the document in which the extracted sentence exists may be extracted and stored together.

The sentence matching unit 120 determines a sentence (first sentence) of the first sentence pool that is matched with a sentence (second sentence) of the second sentence pool based on a preset similarity.

Matching a sentence means checking whether both sentences match each other, and a commonly used text matching technique may be used.

Similarity refers to the degree of similarity between the two sentences, and 100% similarity refers to the case where the letters of both sentences are identical except for spacing and punctuation.

For example, in the second sentence, “When I was six years old, I saw a wonderful picture in a book about primitive forests, titled <Experience>,” when I converted the text from human-reading speech data back into text, When I was living, I saw a wonderful picture in a book about the primeval forest titled Experience Story. ”It could be the first sentence. (The underline is shown to show the difference from the second sentence.)

In this case, the two sentences are not 100% identical, but it is correct to determine the matching sentences, and it is necessary to set the similarity for discriminating both sentences as matching sentences in consideration of the speech recognition degree of the voice data.

The sentence matching unit 120 may adjust the second sentence based on the first sentence when the matched length of the first sentence and the second sentence is different.

That is, when the second sentence is extracted from the document data, semantic analysis, grammar analysis, and punctuation of the text sentence are used, but since the extracted sentence cannot be grammatically perfect, it is based on the speech data read by humans. It is to correct the extracted sentence.

The adjusting of the second sentence may include splitting the second sentence based on the end of the first sentence when the first sentence is shorter than the second sentence, and re-adding a part after the end of the second sentence to the second sentence. Can be stored as a 2 sentence pool.

Further, when the first sentence is longer than the second sentence, the next sentence (third sentence) to be matched in the second sentence pool is divided based on the end of the first sentence, and until the end of the third sentence. The part of may be added to the end of the second sentence, and the part after the end may be stored back into the second sentence pool.

For example, the first sentence says, “When I was six years old, I saw a wonderful picture in a book on the primeval forest titled Experience Story.” The second sentence says, “When I was six years old, I saw a nice picture. The picture was about a boa constrictor. ”Based on the end of the first sentence,“ I saw, ”the later part of the second sentence“ The picture was about a boa constrictor. ” Separate and store as second sentence pool again.

In another example, the first sentence says, “When I was six years old, I saw a nice picture in a book on the primeval forest titled Experience Story, which was about a boa constrictor.” I saw a nice picture in a book on the primeval forest. ”, The next sentence in the second sentence pool (third sentence)“ The picture was about a boa constrictor. ” Based on the above, the first sentence is separated from the third sentence and added to the end of the second sentence. Thus, the second sentence could say, "When I was six years old, I saw a nice picture in a book about the primeval forest titled Experience Story, which was about a boa constrictor."

When a second sentence having a similarity or higher match with the first sentence is determined, the synchronization unit 130 synchronizes the time information of the first sentence with the second sentence and stores the same.

Synchronization means that when a specific sentence is selected from the document data, the time sentence in which the voice sentence reading the sentence is located in the voice data is known.

For example, the time information of the first sentence may be stored together with the second sentence in document data received as additional meta information of the second sentence, or may be stored together with the second sentence in a separate file.

In addition, the synchronization unit 130 may synchronize time information of the first sentence with sentence position information of the second sentence.

For example, if the second sentence is located at 100 mm to the right and 150 mm to the right with the left edge as the origin on page 4 of the document data, the position information of the second sentence may be expressed as “Page 4: X_100, Y_150”. Can be. In addition, the location information of the second sentence may be stored in a separate file together with the time information of the first sentence (eg, “00:04:00”).

The controller 140 performs matching and synchronization until there are no sentences to match in either the first sentence pool or the second sentence pool, thereby completing matching and synchronization of the received voice data and document data.

An embodiment of an e-book authoring method of an e-book authoring apparatus according to the present invention according to FIG. 2 will be described. According to FIG. 2, the e-book authoring method of the e-book authoring apparatus of the present invention comprises the steps of: (a) receiving voice data and document data, (b) processing the voice data and the document data, respectively, and pooling a first sentence together; (Pool) and storing in the second sentence pool, (c) determining a sentence (first sentence) of the first sentence pool matching the sentence (second sentence) of the second sentence pool (d) synchronizing and storing the time code of the first sentence with the second sentence and (e) repeating steps (c) and (d) until there are no sentences to match It includes a step.

The e-book authoring apparatus receives the voice data and the document data (S210), and recognizes the information of the voice data with respect to the input data, converts it into text in a sentence unit, and then stores time information about the voice data ( Time code) is stored in the first sentence pool, and text is extracted from the document data in sentence units and stored in the second sentence pool (S220).

Speech data may be divided into sentence units through semantic analysis and grammatical analysis of text sentences, or sentence units of voice data may be classified by simultaneously analyzing a blank length and sentence meaning analysis.

The data converted into text in sentence units is stored in the first sentence pool along with time information of the voice data.

The time information refers to information on a time at which voice data corresponding to the sentence conversion data is located in the entire voice data.

When extracting document data as text in sentence units, it is possible to use semantic analysis and grammatical analysis of text sentences, or to simultaneously use punctuation marks present in document data.

In addition, when text is extracted from the document data in sentence units, sentence position information of the extracted sentence with respect to the document data may also be extracted and stored as the second sentence pool. That is, up, down, left, and right coordinates of the corresponding page of the document in which the extracted sentence exists may be extracted and stored together.

After the extraction of the sentences from the input voice data and the document data, the sentences of the first sentence pool (first sentence) that are matched with the sentences (second sentence) of the second sentence pool based on a preset similarity ) Is determined (S230).

Matching a sentence means checking whether both sentences match each other, and a commonly used text matching technique may be used.

Similarity refers to the degree of similarity between two sentences, and 100% means that the letters of both sentences are identical except for spacing and punctuation.

The second sentence may be adjusted based on the first sentence when the length of the matched first sentence and the second sentence is different after the determination of the matched sentence.

The adjusting of the second sentence may include: when the first sentence is shorter than the second sentence, dividing the second sentence based on the end of the first sentence, and repeating the part after the end of the second sentence. Can be stored as a second sentence pool.

In addition, when the first sentence is longer than the second sentence, the next sentence (third sentence) to be matched in the second sentence pool is divided based on the end of the first sentence, and thus the end of the third sentence. The part up to may be added to the end of the second sentence, and the part after the end may be stored back into the second sentence pool.

When a second sentence having a similarity or higher match with the first sentence is determined, time information of the first sentence is synchronized with the second sentence and then stored (S240).

Also, time information of the first sentence may be stored after being synchronized with sentence position information of the second sentence.

Synchronization means that when a specific sentence is selected from the document data, the time sentence in which the voice sentence reading the sentence is located in the voice data is known.

In any one of the first sentence pool and the second sentence pool, the matching and synchronizing steps are repeatedly performed until there are no sentences to match (S250) to complete matching and synchronization of the received voice data and document data.

The specific terminology used in the illustrated embodiments is not intended to limit the invention in order to facilitate understanding of the present invention. The present invention may include all components and equivalents of all components that are of ordinary skill in the art.

Claims (8)

(a) receiving audio data and document data, respectively;
(b) converting the information of the voice data into a text in a sentence unit by recognizing the voice, and storing the information in the first sentence pool along with the time code of the voice data, and storing the document data. Extracting text from a sentence unit and storing the extracted text as a second sentence pool;
(c) determining a sentence (first sentence) of the first sentence pool that matches the sentence (second sentence) of the second sentence pool based on a preset similarity;
(d) synchronizing and storing time information of the first sentence in synchronization with the second sentence; And
(e) repeating steps (c) and (d) in either the first sentence pool or the second sentence pool until there are no sentences to match;
, &Lt; / RTI &
After step (c),
(c-2) adjusting the second sentence based on the first sentence when the length of the first sentence and the second sentence are different;
E-book authoring method of the e-book authoring device further comprising
The method according to claim 1,
When the text is extracted from the document data in units of sentences in step (b), sentence position information of the extracted sentence with respect to the document data is also extracted and stored as a second sentence pool. EBook authoring method
delete 3. The method of claim 2,
In the step (d), the synchronization synchronizes the time information with the sentence position information of the second sentence.
Receives voice data and document data, respectively, and recognizes the voice data information and converts the voice data into text in a sentence unit, and then first sentence pool together with time code of the voice data. A sentence extracting unit configured to extract the text from the document data in sentence units and store the extracted text as a second sentence pool;
A sentence matching unit determining a sentence (first sentence) of the first sentence pool that is matched with a sentence (second sentence) of the second sentence pool based on a preset similarity;
A synchronization unit configured to store time information of the first sentence in synchronization with the second sentence;
A memory unit for storing the first sentence pool, the second sentence pool, and synchronization information; And
A controller configured to control matching and synchronization through the sentence matching unit and the synchronization unit until there are no sentences to match in any one of the first sentence pool and the second sentence pool;
, &Lt; / RTI &
The sentence matching unit adjusts the second sentence based on the first sentence when the length of the matched first sentence and the second sentence is different.
6. The method of claim 5,
The sentence extracting unit extracts the sentence position information of the extracted sentence with respect to the document data when extracting text from the document data, and stores the second book as a second sentence pool.
delete The method according to claim 6,
The synchronizing unit is an e-book authoring apparatus, characterized in that for synchronizing the time information (Time Code) with the sentence position information of the second sentence

Images