KR102541162B1 - Electronic apparatus and methoth for caption synchronization of contents - Google Patents

Abstract

An electronic device is disclosed. The present electronic device extracts voice recognition data and time information by performing voice recognition on image content and a communication device receiving caption data for the image content, memory, and image content, and character strings and captions of the extracted voice recognition data. a processor for sorting data strings, modifying time information in the caption data with the extracted time information based on the sorting result, and storing the caption data with the corrected time information in a memory; The similarity between the text string and the subtitle data is calculated, and the text string of the speech recognition data and the text string of the caption data are aligned based on the interval score in word and/or sentence units and the calculated similarity.

Description

Electronic device and method for performing subtitle synchronization for content

The present disclosure relates to an electronic device and method for performing caption synchronization on content, and more particularly, to an electronic device and method for performing caption synchronization on content with high accuracy.

Domestic terrestrial broadcasting provides closed-captioning services for the hearing-impaired over the entire broadcasting time in accordance with relevant laws and regulations. Currently, closed captioning is produced in the form of a stenographer inputting captions in real time, sending them back to broadcasting companies, and inserting them into terrestrial broadcasting.

Recently, broadcasting companies are processing corresponding broadcast contents after terrestrial broadcasting and using them for online services. During this process, a time offset is generated due to pre-production, advertisement insertion, and the like, and accordingly, the start time of terrestrial broadcasting and the start time of processed broadcasting content may be different.

However, since subtitles for terrestrial broadcasting (hereinafter referred to as closed captions) are stored in the system as they are without additional editing, there is a problem in that synchronization is out of sync when the corresponding subtitles are used in an online service.

Accordingly, recently, a method of generating voice recognition text using a voice recognition technology and comparing the generated voice recognition text with the closed caption to correct a sync in the closed caption has been proposed.

However, as the conventional method compares text only in units of syllables, there is a problem in that syncs are separated within one sentence or within words within closed captions.

Accordingly, an object of the present disclosure is to provide an electronic device and method for synchronizing subtitles to content with high accuracy.

An electronic device according to the present disclosure for achieving the above object is, in the electronic device, a communication device for receiving video content and subtitle data for the video content, a memory, and voice recognition for the video content to provide audio Recognition data and time information are extracted, a character string of the extracted speech recognition data and a character string of the caption data are aligned, and based on a result of the alignment, time information in the caption data is modified to the extracted time information, and time information is modified. and a processor for storing caption data with corrected information in the memory, wherein the processor calculates a similarity between a character string of the voice recognition data and a character string of the caption data, and calculates a spacing score in units of words and/or sentences and the calculation The character string of the voice recognition data and the character string of the subtitle data are aligned based on the similarity.

In this case, the processor divides the character string in the subtitle data into sentence units, and assigns a first interval score of a code opposite to the similarity to a character string spacing in the divided sentence, so that the character string of the speech recognition data and the caption Strings of data can be sorted.

In this case, the processor may detect punctuation marks in the caption data, and divide the character string in the caption data into sentence units based on the detected punctuation marks.

Meanwhile, the processor may align the character strings of the speech recognition data and the character strings of the subtitle data by assigning a second spacing score having the same code to the similarity for the spacing between the divided sentences.

Meanwhile, the processor divides each of the character string in the caption data and the character string of the caption data in word units, and assigns a third interval score of a code opposite to the similarity to the character string spacing in the separated words, thereby generating the voice A character string of the recognition data and a character string of the caption data may be aligned.

Meanwhile, the processor may determine a search area corresponding to the number of characters of the caption data, and calculate a similarity between a character string of the voice recognition data and a character string of the caption data within the determined search area.

Meanwhile, the processor may calculate the degree of similarity based on whether initial consonants, neutral consonants, and final consonants are identical within a syllable unit.

In this case, the processor calculates a first value if the initial consonant, neutral consonant, and final consonant are all the same within the syllable unit, and calculates a second value smaller than the first value if one of the initial consonant, neutral consonant, and final consonant is different, and If only one of the final consonants is the same, a third value smaller than the second value is calculated, and if all of the initial consonant, neutral consonant, and final consonant do not match, a fourth value smaller than the third value is calculated, and the calculated value of the calculated syllable unit A character string of the voice recognition data and a character string of the caption data may be aligned based on the interval score.

In this case, the first to fourth values may have nonlinear values.

Meanwhile, the processor may modify time information on the first syllable of the line of the caption data to time information on the text string of voice recognition data corresponding to the text string based on the alignment result.

In this case, if there is no character string in the caption data corresponding to the first character string of the line of the caption data, the processor uses time information on the character string of voice recognition data corresponding to the character string adjacent to the first character string and a preset speech time. Thus, time information on the first string may be estimated, and time information on the first string may be corrected to the estimated time information.

In this case, the processor may extract a character string excluding text auxiliary information corresponding to the context description of the video from the received caption data.

Meanwhile, the caption data for the video content may be a commentary caption transmitted in the form of an electronic code corresponding to the video content.

Meanwhile, the processor may control the communication device to transmit the modified caption data to an external device.

Meanwhile, a method for synchronizing captions of an electronic device according to an embodiment of the present disclosure includes receiving video content and caption data for the video content, performing voice recognition on the video content, and extracting voice recognition data and time information. aligning the extracted voice recognition data string with the caption data string, correcting time information in the caption data based on the sorting result, and storing the caption data with corrected time information. and the step of sorting calculates a similarity between a character string of the voice recognition data and a character string of the subtitle data, and calculates a similarity between a character string of the voice recognition data and a character string of the subtitle data, based on a spacing score in units of words and/or sentences and the calculated similarity. A character string and the character string of the subtitle data are aligned.

As described above, according to various embodiments of the present disclosure, it is possible to automatically synchronize subtitle data using a result of voice recognition for video content. In particular, by performing alignment to suppress occurrence of gaps within sentences and/or words, it is possible to synchronize subtitle data with high accuracy.

1 is a diagram showing the configuration of a content creation system according to an embodiment of the present disclosure;
2 is a diagram showing a specific configuration of an electronic device according to an embodiment of the present disclosure;
3 is a diagram for explaining a dynamic programming method according to an embodiment of the present disclosure;
4 is a diagram for explaining a restricted area according to an embodiment of the present disclosure;
5 is a diagram for explaining the relationship between content and a restricted area;
6 is a diagram showing an example of an estimated time code according to an embodiment of the present disclosure;
7 is a diagram showing an example of time code estimation according to the first embodiment;
8 is a diagram showing an example of time code estimation according to the second embodiment;
9 is a view comparing conventional alignment results and alignment results according to an embodiment of the present disclosure;
10 is a flowchart illustrating a caption synchronization method according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

Embodiments of the present disclosure may apply various transformations and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of technology disclosed. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. Terms are only used to distinguish one component from another.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, "comprising." or "made up." The terms such as are intended to specify that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features or numbers, steps, operations, components, parts, or It should be understood that it does not preclude the possibility of existence or addition of combinations thereof.

In an embodiment of the present disclosure, a 'module' or 'unit' performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a diagram showing the configuration of a content creation system according to an embodiment of the present disclosure.

Referring to FIG. 1 , a content creation system 1000 according to an embodiment of the present disclosure may include a broadcast transmitting device 10, a caption generating device 20, a server 30, and an electronic device 100. .

The broadcast transmitting device 10 may transmit terrestrial broadcasting using video content. In this case, the broadcast transmitting device 10 may provide a caption broadcasting service by using caption data for video content generated by the caption generating device 20 . Here, the closed captioning service is a service that displays text corresponding to a voice signal as subtitles on a TV screen. A technology that converts and transmits auditory messages of TV programs in the form of electronic codes for the hearing impaired to display them as commentary subtitles on the TV screen. am.

In addition, the broadcast transmitting device 10 may transmit the video content for which terrestrial broadcasting has been transmitted to the server 30 . At this time, the broadcast transmission device 40 may transmit the video content as it is to the server 30, and may perform an operation such as additionally inserting an advertisement or dividing it into a plurality of pieces, that is, transmit the edited content to the server 30. may be

The caption generating device 20 may generate caption data for video content transmitted by the broadcast transmission device 40 and transmit the generated caption data to the server 30 and/or the broadcast transmission device 40 . Here, the caption data is data generated through a stenographer's typing, and may be a commentary caption transmitted in the form of an electronic code corresponding to video content. Since such subtitle data is generated by the input of a professional stenographer, it has high accuracy, but since it takes time to input the stenography, there is a time offset from video content. In addition, caption data may generally include not only human speech, but also text supplementary information for explaining the situation of a video for explaining vehicle horns, cell phone sounds, and the like. Caption data generated by such a stenographer may be referred to as a closed caption.

The server 30 may store video content transmitted from the broadcast transmitting device 10 and may store caption data for corresponding video content from the caption generating device 20 . Such a server 30 may be an archive server.

The electronic device 100 may receive caption data from the server 30 and correct a sync of the received caption data. In addition, the electronic device 100 may transmit the modified caption data to the server 30 .

At this time, the electronic device 100 receives video content corresponding to the caption data from the server 30, performs speech to text on the received video content, and text to the voice included in the video content. Speech recognition data having time information (or time synchronization information, time code) of each information and text can be generated. Such voice recognition data may be referred to as voice recognition captions, voice recognition results, and the like.

When the voice recognition data is generated, the electronic device 100 compares the caption data with the extracted voice recognition data to correct time information in the caption data to time information in the voice recognition data. A detailed configuration and operation of the electronic device 100 will be described later with reference to FIG. 2 .

As described above, the content generation system 1000 according to the present embodiment corrects time information in caption data using voice recognition data generated through voice recognition of video content, and automatically synchronizes caption data without human intervention. can be fitted to the video.

On the other hand, in the illustration and description of Figure 1, each device is shown in the form of being directly connected to each other, but in the case of implementation, each component may be connected in a form via a separate external component. Also, although the electronic device 100 has been described as being connected to only one server 30, the electronic device 100 can be connected to a plurality of servers and can individually receive video content and subtitle data from each server. .

In addition, in the illustration and description of FIG. 1, the server 30 and the electronic device 100 are shown and described as being separate, but in implementation, the two devices may be implemented as one. That is, the server 30 that stores the video content may perform synchronizing the caption data by itself without transmitting the video content to the electronic device 100 and synchronizing captions.

2 is a diagram illustrating a specific configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 may include a communication device 110, a memory 120, a display 130, a manipulation input device 140, and a processor 150. Here, the electronic device 100 may be a PC capable of image processing, a notebook PC, a smart phone, a server, and the like.

The communication device 110 is connected to the caption generating device 20 and/or the server 30 and can transmit and receive video content and caption data. Specifically, the communication device 110 is formed to connect the electronic device 100 with an external device, and is connected to a mobile device through a Local Area Network (LAN) and an Internet network, as well as a Universal Serial Bus (USB) device. It is also possible to connect through a serial bus port.

In addition, the communication device 110 may be connected to other electronic devices via a router or a router connected to a public Internet network as well as a wired method, and the router or router may be connected to another electronic device through a wired method as well as Wi-Fi, Bluetooth, cellular communication, etc. It can also be connected wirelessly.

Also, the communication device 110 may transmit caption data (ie, synchronized caption data) having time information corrected by a process described below to an external device (eg, the server 30).

The memory 120 is a component for storing O/S for driving the electronic device 100, software for performing voice recognition/caption synchronization, data, and the like. The memory 120 may be implemented in various forms such as RAM, ROM, flash memory, HDD, external memory, memory card, etc., but is not limited to any one.

The memory 120 stores video content. Here, the video content includes audio as well as video, and may be a video file such as MP4, AVI, or MOV. On the other hand, the above-described file format is not limited to one example, and is not limited to the above-described examples.

Also, the memory 120 stores caption data corresponding to the above-described video content. Here, the caption data may include character string information to be displayed on the caption service and time information to display the corresponding character string information. Here, the time information may be time information in ms units.

Also, the memory 120 may store character string information, time information, similarity information, etc. generated during a process to be described later, and may store caption data having corrected time information.

The display 130 displays a user interface window for selecting a function supported by the electronic device 100 . Specifically, the display 130 may display a user interface window for selecting various functions provided by the electronic device 100 . The display 130 may be a monitor such as an LCD, a CRT, or an OLED, and may be implemented as a touch screen capable of simultaneously performing the functions of the manipulation input device 140 to be described later.

The manipulation input device 140 may receive a function selection of the electronic device 100 and a control command for the function from the user. Specifically, the manipulation input device 140 may receive a command on whether or not to correct the sync of the received caption data. Also, the manipulation input device 140 may receive selection of caption data (or video content) to perform caption synchronization.

The processor 150 controls each component in the electronic device 100 . Specifically, the processor 150 may perform booting using the operating system stored in the memory 120 when a booting command is input from the user.

When video content or caption data for performing caption synchronization is selected, the processor 150 may control the communication device 110 to receive caption data corresponding to the selected video content or video content corresponding to the selected caption data. .

When video content and caption data corresponding to the video content are received, the processor 150 may perform voice recognition on the video content to extract voice recognition data and time information. For example, the processor 150 may perform speech recognition using any one of well-known libraries related to speech recognition (eg, Kaldi, CMU Sphinx, HTK, AWS Transcribe).

In addition, the processor 150 may compare the extracted voice recognition data and the caption data to correct time information in the caption data to the extracted time information. Specifically, the processor 150 may extract only a character string from previously extracted voice recognition data. For example, the voice recognition data may include a character string and time information of the corresponding character string as described above, and for a process to be described later, the processor 150 may extract only the character string from the voice recognition data.

For example, caption data may include symbols such as '?' and '!', but in many cases, such symbols are not accurately included in text through voice recognition. Accordingly, the processor 150 may extract only character strings excluding the above-described symbols from the received caption data. In addition, the caption data may include text such as information (text supplementary information) for explaining a situation (eg, a phone rings, a car horn sounds). However, since such text supplementary information is information that is not extracted in the voice recognition process, the processor 150 may extract a character string excluding the text supplementary information from the received subtitle data. For example, in the text auxiliary information described above, there is a case in which a character string is described in brackets of "(", ")". Accordingly, the processor 150 may extract a string excluding the string in parentheses from the received subtitle data.

Further, the processor 150 may divide the character string in the caption data into sentence units based on punctuation marks in the received caption data. For example, since closing punctuation marks such as '.', '!', and ?' are marks that complete a sentence, a character string between these marks can be divided into one sentence. On the other hand, punctuation marks such as ',' indicate that sentences are connected, and such punctuation marks may not be used to distinguish sentence units.

Also, the processor 150 may classify the character string of the voice recognition data in units of words. Hereinafter, in order to facilitate explanation, the above-mentioned words refer to not only words that can be separated and used independently, but also words that are combined with a corresponding word and an article indicating a grammatical function of the corresponding word. Generally, in Korean and English, spaces are used to separate words. In this respect, the processor 150 may check areas for spacing in the text string of the voice recognition data and classify the text string between the spaces as a single word.

Also, the processor 150 may align the character string of the voice recognition data with the character string of the caption data. Specifically, the processor 150 may align a character string of voice recognition data and a character string of caption data using a dynamic programming method. Here, the dynamic programming method is a task of aligning two long strings, and is widely used in analyzing genome sequences in the field of bioinformatics. There are many algorithms for dynamic programming, but the Needleman-Wunsch algorithm was used in the present disclosure, but is not limited thereto. The method using the Needleman-Wunsch algorithm will be described in detail with reference to FIG. 5 .

On the other hand, existing dynamic programming methods compare similarity by simply considering only character strings, so strings within one sentence and/or word can be sorted as having high similarities to different objects. When such alignment occurs, it is necessary to prevent separation of strings within one sentence and/or word in that text within one sentence may be displayed in different sinks.

To this end, in the present disclosure, a character string of voice recognition data and a character string of subtitle data are aligned using not only similarities between character strings but also interval scores in units of words and/or sentences. A detailed method according to this will be described later with reference to FIG. 3 .

Further, the processor 150 may modify time information in caption data to extracted time information based on the alignment result. Specifically, the processor 150 may modify time information on the start point (ie, the first syllable) of a line of caption data to time information on a text string of voice recognition data corresponding to the corresponding text string based on the sorting result. On the other hand, when there is no character string of voice recognition data corresponding to the starting point of the corresponding line, the processor 150 uses time information on the character string of voice recognition data corresponding to the line before or after the corresponding line to determine the time of the corresponding line. Information may be estimated, and time information of a corresponding line may be corrected with the estimated time information. Such an operation will be described later with reference to FIGS. 6 to 8 .

Also, the processor 150 may store the closed caption data with time information corrected in the memory 120 . Meanwhile, when caption data is stored in an external device (eg, the server 30) and provided to other devices, the processor 150 controls the communication device 110 to transmit the modified caption data to the external device. can do.

Meanwhile, although the electronic device 100 has been described above as correcting (or updating) time information of caption data, the above-described operation generates new caption data by combining character string information of caption data and time information of voice recognition data. It can also be expressed as

As described above, the electronic device 100 according to the present embodiment corrects time information in caption data using voice recognition data generated through voice recognition of video content, thereby synchronizing caption data with video data more easily and quickly. it is possible In particular, by performing alignment to suppress occurrence of gaps within sentences and/or words, it is possible to synchronize subtitle data with high accuracy.

Meanwhile, in the drawings and descriptions of FIGS. 1 and 2, although the above-described operation is shown and described as being performed only for content generated by a broadcasting station, separate subtitle data input by a stenographer also applies to video content generated by a station other than a broadcasting station. If there is, it can be applied.

On the other hand, in the illustration and description of FIG. 2, although the electronic device 100 has been illustrated and described as including a display and a manipulation input device, when the electronic device 100 is implemented as a device such as a server, the above-described display and A manipulation input device may be omitted.

Hereinafter, a dynamic programming method according to the present disclosure will be described.

Due to the development of voice recognition technology, voice recognition technology for video content has evolved considerably. Since voice recognition is performed on video content, the time synchronization information for the calculated text is very accurate. However, since the accuracy of voice recognition is still lower than that of the caption directly input by the stenographer, it is difficult to provide the caption only with the voice recognition result.

However, closed captions input by a stenographer have high accuracy in terms of content, but offsets occur due to processes such as storage of closed captions, broadcasting production and archiving processes, etc., so time synchronization information in closed captions is different from actual video. Advantages and disadvantages of voice recognition captions and closed captions can be summarized as shown in Table 1 below.

text information time synchronization information closed captions exact inaccuracy voice recognition subtitles inaccuracy exact

Accordingly, in the present disclosure, caption data having high accuracy can be generated by using the text information of the closed caption and synchronizing the time synchronization information with the time information of the speech recognition caption.

Meanwhile, in order to compare text of closed caption and text of voice recognition, the present disclosure uses dynamic programming.

Dynamic programming refers to a method in which, when a problem is composed of repeated subproblems in several steps, the solution to the entire problem is obtained based on the answers to the subproblems in each step. This dynamic programming method is a method of solving from the smallest subproblem to the upper problem.

This dynamic programming method is suitable for aligning general strings such as base sequences, but additional constraints need to be considered in the problem of synchronizing subtitles.

First, in general sequence alignment, such as base sequences, when each element is not matched, insertion and deletion occur, which are called gaps.

In the caption synchronization problem, a closed caption string can be divided into sentence units (or word units) and a speech recognition string can be divided into word units, and it is an additional constraint that it is difficult to generate large gaps therein.

In general, there are two cases where large gaps occur in TV programs. The first case is when the contents of speech are displayed as subtitles on the screen. When a caption on the screen is displayed, the corresponding text is not included in the closed caption. On the other hand, since text is generated according to the contents of speech in the speech recognition result, a large gap may occur.

The second is for songs that do not have voice recognition. In the case of a song, text corresponding to the lyrics of the song exists on the closed caption, but since voice recognition is not performed for the song and text is not generated in the voice recognition process, a large gap may occur.

As such, in the present disclosure, alignment is performed in consideration of the above-described additional constraints.

3 is a diagram for explaining a dynamic programming method according to an embodiment of the present disclosure. Specifically, FIG. 3 shows a similarity matrix 310 when the above-described additional constraint conditions are not reflected and a similarity matrix 320 when the above-described additional constraint conditions are reflected.

First, the sorting operation without reflecting the additional constraints will be described first, and then the sorting operation in the case of reflecting the additional constraints will be described.

Referring to FIG. 3 , in the first similarity matrix 310, character strings of closed captions are disposed in the horizontal direction, and character strings of speech recognition results are disposed in the vertical direction.

In order to apply the dynamic programming method to a character string, the smallest part of the character string can be used as a syllable, that is, the similarity can be calculated in units of syllables.

As such, the similarity in syllable units is first compared with two syllables, and if they are identical, the similarity is calculated as a first value (+1), and if they are not identical or there is information missing, a method of calculating the similarity as a second value (-1) can be used

However, when comparing only the same or mismatched cases, if the syllable form is different even with similar pronunciations, such as 'gong'-'bong', the above method may result in performance degradation because it is judged as a mismatch.

Therefore, after decomposing a syllable into phonemes in initial/middle/final consonant units, the degree of similarity may be calculated by comparing each phoneme in initial/middle/final consonant units. For example, the initial/neutral/final of each syllable is compared, and if all phonemes are the same, a first value (eg, 38), and if one of the initial/neutral/final is different, a second value (eg, 19), an initial consonant If only one of the /neutral/final consonant is the same, a third value (eg, 3) may be calculated, and if all of the initial/neutral/final consonants are different, a fourth value (eg, 0) may be calculated. In this way, each value may be given non-linearly.

On the other hand, in the above, only the method of giving +38/+19/+3/0 points in phoneme units has been described, but in implementation, the above-described values are not illustrative, and values different from the numerical values above may be used, The similarity value may be calculated by assigning different weight values to the initial, middle, and final sexes.

When the similarity of each syllable unit is calculated, the two character strings can be sorted by finding a path having the maximum cumulative similarity value using the calculated similarity.

A path found through such an operation is the same as a solid line in the similarity matrix 310 .

On the other hand, looking at the middle region 311 of the similarity matrix 310, it is possible to check a section falling at a certain slope. Such a section is an area where a gap may occur between character strings within a sentence in one closed caption. For example, with respect to one sentence of “News is delivered” in the closed caption, the part “News” and “We will deliver” may be mapped to different regions spaced apart in time.

In this way, if one sentence in the closed caption is mapped to a spaced character string in the speech recognition result, sync accuracy is not matched.

Accordingly, alignment may be performed by additionally assigning a spacing score that minimizes spacing in one sentence (or word) in the closed caption and minimizes spacing in words in the speech recognition result.

For this sorting, it is necessary to know in advance how sentences in the closed caption are classified and how words in the text of the speech recognition result are classified in advance. Sentences can be distinguished by detecting punctuation marks in closed captions as described above, and words can be distinguished by using spaces. A similarity matrix reflecting such division is shown as 320 in FIG. 3 .

In the second similarity matrix 320, character strings of closed captions are disposed in the horizontal direction, and character strings of voice recognition results are disposed in the vertical direction. Blue vertical lines indicate sentence spacing of closed captions, and red horizontal lines indicate word spacing of speech recognition subtitles.

In this way, by dividing the character string in the subtitle data into sentence units, a first spacing score of a code opposite to the degree of similarity may be given to the spacing of the character string in the divided sentence. Conversely, for spacing between sentences, a spacing score having the same sign in similarity can be assigned. In addition, it is possible to divide the character string in the caption data into word units or divide the character string in the voice recognition result into word units to give interval scores in a similar manner as described above.

In this way, when the syllable-unit similarity and interval scores are calculated, the two character strings can be sorted by finding a path having the maximum cumulative similarity value using the calculated similarity and interval scores.

When the second similarity matrix 320 calculated by reflecting the spacing scores as described above is compared with the first similarity matrix 310 (specifically, in the middle regions 311 and 312, gaps occur in sentences within closed captions). decreases, and it can be confirmed that the gap difference occurs only between sentences.

Table 2 below summarizes the above-mentioned syllable-based scoring system and the scoring system that reflects additional restrictions.

type value explanation Match
scores 0 syllable mismatch 3 match one syllable 19 match two syllables 38 match all syllables gap
scores -0.9 Gaps within words within closed captions -15 Gaps within sentences of closed captions -0.5 Gaps within words within speech recognition results 0 Gaps between words in speech recognition results 3 Gap after end of sentence in closed caption

In the above table, four items were used for the gap score, but in implementation, other items other than the above items may be used, or only some of the above four items may be used. In addition, the above numerical values are merely examples, and other numerical values may be used depending on the type of content and system environment.

Meanwhile, for the above-described string sorting, similarity calculations must be performed for all strings in two texts. In particular, since the calculation time increases in proportion to the square of the number of letters to be sorted, it is necessary to reduce the time required to calculate the similarity.

A method of reducing the calculation time will be described below with reference to FIGS. 4 and 5 .

4 is a diagram for explaining a restricted area according to an embodiment of the present disclosure. 5 is a diagram for explaining the relationship between content and a restricted area.

Referring to FIG. 4 , a search area (or restriction area) for performing similarity calculation is set.

As shown in FIG. 3 , the path having the maximum cumulative similarity value passes through the center area of the similarity matrices 310 and 320 . Therefore, even if the similarity calculation is not performed for the upper right and lower left regions except for the central region, the route calculation is not affected.

However, if the central area to be searched is narrowed, the calculation speed is improved, but the optimal size of the central area is determined in that accuracy decreases when the path with the maximum cumulative similarity value deviates from the central area. There is a need.

To this end, as a result of analyzing various contents, the correlation between the length of the closed caption and the size of the search area was confirmed as shown in FIG. 5 . Specifically, the number of characters of the closed caption and the size of the search area may be modeled as a logarithmic function. Accordingly, when a closed caption is input, the electronic device 100 checks the number of characters of the input closed caption, determines a search area corresponding to the checked number of characters, and generates the above-described similarity matrix only for the determined search area. can

If it is desired to reduce the memory used in the constrained dynamic programming method, the black area (ie, the restricted area) of FIG. 4 may be removed, and only the search area (green area) may be reconstructed into a rectangular shape and used.

As described above, after the text of the caption data and the voice recognition data are aligned, time information in the closed caption may be corrected using the alignment result.

Specifically, the time information may be corrected by modifying the time information of the first syllable of the line (or sentence) of the caption data to the time information of the character string of voice recognition data corresponding to the character string.

However, if there is no character string of voice recognition data corresponding to the first syllable of a line of caption data, time information cannot be modified in the above-described manner.

For example, as exemplified above, in the case of a song, if there is a character string in the closed caption, but there is no character string corresponding to the voice recognition data, time information of the voice recognition data cannot be used.

Hereinafter, a method of correcting time information in this case will be described.

6 is a diagram showing an example of an estimated time code according to an embodiment of the present disclosure, FIG. 7 is a diagram showing an example of time code estimation according to the first embodiment, and FIG. 8 is a diagram showing an example of time code estimation according to the second embodiment. It is a diagram showing an example of time code estimation.

First, linear regression analysis can be performed with time codes of matched letters in the sorting process to estimate time information of unmatched letters. To this end, in cases other than the first letter of a word, the time code for each letter can be estimated using the total number of letters and the utterance time. For example, the time code may be estimated using a value such as (SR/start) shown in FIG. 6 .

At this time, since inaccurate estimation can be made due to a small number of outliers as shown in FIG. 7 with only simple least squares based linear regression analysis, as shown in FIG. Thus, the influence of outliers can be minimized.

On the other hand, if more than a certain number of matches do not occur, time code estimation by regression analysis is abandoned, and the time code can be recovered by searching in the forward and backward directions of a line having an accurate time code and referring to it.

9 is a diagram comparing a conventional alignment result and an alignment result according to an embodiment of the present disclosure.

Referring to FIG. 9 , an alignment result 910 when the additional constraint conditions are not reflected and an alignment result 920 when the additional constraint conditions are reflected are shown.

Comparing the two sorting results, if the existing additional constraints are not reflected, the character string in one sentence in the closed caption (CC) is mapped to the spaced character string in the speech recognition result. When the additional constraint conditions are reflected, it can be seen that the spacing of the character strings in the closed caption is limited, and the synchronization accuracy is greatly improved.

10 is a flowchart illustrating a caption synchronization method according to an embodiment of the present disclosure.

Referring to FIG. 10 , first, video content and caption data for the video content are received (S1010). For example, if video content or caption data that is a subject of caption synchronization is selected, the selected video content or caption data may be received, and caption data or video content corresponding to the received video content or received caption data may be searched for and received. there is. Meanwhile, at the time of implementation, at least one of video content or subtitle data may be stored in advance, and in this case, the above-described receiving operation may be omitted.

Then, voice recognition is performed on the video content to extract voice recognition data and time information (S1020). Specifically, voice recognition data having text and time information of the text may be generated by recognizing the voice of video content using a voice recognition library.

The character string of the extracted voice recognition data and the character string of the caption data are aligned (S1030). Specifically, the similarity between the character string of the voice recognition data and the character string of the caption data may be calculated, and the character string of the voice recognition data and the character string of the caption data may be sorted based on the interval score of each word and/or sentence and the calculated similarity. .

Then, based on the alignment result, time information in the caption data is corrected (S1040). Specifically, time information in caption data may be corrected to extracted time information based on a result of the alignment. For example, if there is a character string in caption data corresponding to the first syllable of a line of caption data, time information of the corresponding syllable may be modified to time information of voice recognition data corresponding to the corresponding syllable. If there is no character string in the caption data corresponding to the first character string of the caption data line, the time for the first character string is determined using the time information of the character string of voice recognition data corresponding to the character string adjacent to the first character string and the preset speech time. Information may be estimated, and the time information for the first character string may be modified to the estimated time information.

Caption data with corrected time information is stored (S1050). If caption data is received from an external device, caption data with corrected time information may be transmitted to the external device.

Accordingly, in the file transmission method according to the present embodiment, time information in caption data is modified using voice recognition data generated through voice recognition of video content, so that caption data can be synchronized with video easily. Also, by performing alignment to suppress occurrence of gaps within sentences and/or words, it is possible to synchronize subtitle data with high accuracy. The caption synchronization method of FIG. 10 can be executed on an electronic device having the configuration shown in FIG. 2 and also on electronic devices having other configurations.

In addition, the above-described subtitle synchronization method may be implemented as a program including an executable algorithm that can be executed on a computer, and the above-described program may be stored and provided in a non-transitory computer readable medium. can

A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, programs for performing the various methods described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

In addition, although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure belongs without departing from the subject matter of the present disclosure claimed in the claims. Of course, various modifications are possible by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

1000: content creation system 10: broadcast transmission device
20: subtitle generating device 30: server
100: electronic device 110: communication device
120: memory 130: display
140: operation input device 150: processor

Claims (15)

In electronic devices,
a communication device that receives video content and caption data for the video content;
Memory; and
Voice recognition is performed on the video content to extract voice recognition data and time information, align a character string of the extracted voice recognition data and a character string of the caption data, and based on the alignment result, time information in the caption data and a processor for correcting the extracted time information and storing closed caption data having the corrected time information in the memory,
the processor,
The similarity between the character string of the voice recognition data and the character string of the subtitle data is calculated based on whether the initial consonants, middle consonants, and final consonants are identical within the syllable unit, and based on the interval score of at least one of the word unit and sentence unit and the calculated similarity An electronic device that aligns a string of the voice recognition data with a string of the caption data. According to claim 1,
the processor,
Dividing the character string in the caption data into sentence units, and assigning a first interval score of a code opposite to the similarity to the character string spacing in the divided sentences to align the character string of the voice recognition data and the character string of the caption data electronic device. According to claim 2,
the processor,
An electronic device that detects punctuation marks in the caption data and divides a character string in the caption data into sentence units based on the detected punctuation marks. According to claim 2,
the processor,
The electronic device aligns the text strings of the speech recognition data and the text strings of the subtitle data by assigning a second spacing score having the same code to the similarity for the spacing between the divided sentences. According to claim 1,
the processor,
A character string in the caption data and each character string of the caption data are divided into word units, and a third spacing score of a code opposite to the similarity is given to the distance between the character strings in the separated words, so as to match the character string of the voice recognition data. An electronic device for arranging text strings of the caption data. According to claim 1,
the processor,
An electronic device that determines a search area corresponding to the number of characters of the caption data, and calculates a similarity between a character string of the voice recognition data and a character string of the caption data within the determined search area. delete According to claim 1,
the processor,
If all of the initial, neutral, and final consonants are the same within the syllable unit, a first value is calculated, and if one of the initial, neutral, and final consonants is different, a second value smaller than the first value is calculated, and if only one of the initial, neutral, and final consonants is the same, the first value is calculated. A third value smaller than the second value is calculated, and a fourth value smaller than the third value is calculated when the initial consonant, neutral consonant, and final consonant do not match,
An electronic device that aligns the text string of the voice recognition data with the text string of the caption data based on the calculated value of the calculated syllable unit and the spacing score. According to claim 8,
The first to fourth values have non-linear values. According to claim 1,
the processor,
An electronic device that modifies time information of a first syllable of a line of the caption data to time information of a character string of speech recognition data corresponding to the character string based on the alignment result. According to claim 10,
the processor,
If there is no character string in the caption data corresponding to the first character string of the line of the caption data, the response to the first character string is determined by using time information on the character string of voice recognition data corresponding to the character string adjacent to the first character string and a preset speech time. An electronic device for estimating time information and correcting time information for the first character string to the estimated time information. According to claim 1,
the processor,
An electronic device for extracting a character string excluding text auxiliary information corresponding to a situation description of an image from the received caption data. According to claim 1,
Subtitle data for the video content,
An electronic device that is a commentary caption transmitted in the form of an electronic code corresponding to the video content. According to claim 1,
the processor,
An electronic device that controls the communication device to transmit the modified caption data to an external device. A method for synchronizing subtitles of an electronic device,
Receiving video content and caption data for the video content;
extracting voice recognition data and time information by performing voice recognition on the video content;
arranging a character string of the extracted voice recognition data and a character string of the caption data;
modifying time information in the caption data based on the sorting result; and
Storing caption data with corrected time information;
The sorting step is
The similarity between the character string of the voice recognition data and the character string of the subtitle data is calculated based on whether the initial consonants, middle consonants, and final consonants are identical within the syllable unit, and based on the interval score of at least one of the word unit and sentence unit and the calculated similarity A caption synchronization method of arranging a character string of the voice recognition data and a character string of the caption data.

Images