KR102385779B1 - Electronic apparatus and methoth for caption synchronization of contents - Google Patents

Abstract

An electronic device is disclosed. The electronic device extracts voice recognition data and time information by performing voice recognition on video content and a communication device that receives video content and caption data for the video content, a memory, and video content, and extracts the extracted speech recognition data and the received caption and a processor that compares the data, corrects temporal information in the subtitle data to the extracted temporal information, and stores the corrected subtitle data in a memory.

Description

ELECTRONIC APPARATUS AND METHOTH FOR CAPTION SYNCHRONIZATION OF CONTENTS

The present disclosure relates to an electronic device and method for performing subtitle synchronization on content, and more particularly, to an electronic device and method for automatically performing subtitle synchronization on content.

In accordance with relevant laws and regulations, domestic terrestrial broadcasting provides closed captioning services for the hearing impaired throughout the entire broadcast time. The current production of closed captioning is done in the form of a stenographer inputting captions in real time, transmitting them back to the broadcaster, and inserting them into terrestrial broadcasts.

Recently, broadcasters are processing the corresponding broadcast content after terrestrial broadcasting and using it for online services. During this process, a time offset occurs due to pre-production, advertisement insertion, etc., and accordingly, the start time of terrestrial broadcasting and the start time of processed broadcast content may be different. However, since subtitles for terrestrial broadcasting (hereinafter, closed subtitles) are stored as they are in the system without additional editing, there is a problem in that when the corresponding subtitles are used in an online service, the synchronization becomes out of sync.

Accordingly, an object of the present disclosure is to provide an electronic device and method for automatically performing subtitle synchronization for content.

In order to achieve the above object, an electronic device according to the present disclosure provides a communication device for receiving image content and caption data for the image content, a memory, and voice recognition for the image content to obtain voice recognition data and time information. a processor configured to extract, compare the extracted speech recognition data with the received caption data, correct time information in the caption data with the extracted time information, and store the caption data with the time information corrected in the memory; include

In this case, the processor extracts only the character string from the extracted voice recognition data, extracts only the character string from the received subtitle data, aligns the character string of the voice recognition data and the character string of the subtitle data, and based on the alignment result Thus, time information in the caption data can be modified with the extracted time information.

In this case, the processor calculates a similarity of the character string of the voice recognition data and the character string of the subtitle data in units of syllables, and based on a syllable having a predetermined similarity among the calculated similarities, the character string of the voice recognition data and the subtitle data Strings can be sorted.

In this case, the processor may calculate the degree of similarity based on whether the initial consonant, the middle consonant, and the final consonant are identical within a syllable unit.

In this case, the processor calculates a first value if all of the leading, neutral, and final consonants are the same within a syllable unit, and calculates a second value that is smaller than the first value if one of the leading, neutral, and final consonants is different, If only one of them is the same, a third value smaller than the second value may be calculated, and the character string of the voice recognition data and the character string of the subtitle data may be arranged based on the syllable having the first value and the second value.

Meanwhile, the processor may align the character string of the voice recognition data and the character string of the subtitle data using dynamic programming and the calculated similarity.

Meanwhile, the processor may modify the time information on the first syllable of the line of the subtitle data to the time information on the character string of the voice recognition data corresponding to the character string based on the alignment result.

Meanwhile, the processor may extract a character string excluding the text auxiliary information corresponding to the contextual description of the image from the received caption data.

Meanwhile, the caption data for the video content may be an explanation caption transmitted in the form of an electronic code corresponding to the video content.

Meanwhile, the processor may control the communication device to transmit the modified caption data to an external device.

Meanwhile, the method for synchronizing captions of an electronic device according to an embodiment of the present disclosure includes receiving video content and caption data for the video content, and performing voice recognition on the video content to extract speech recognition data and time information. comparing the extracted speech recognition data with the received caption data to correct time information in the caption data with the extracted time information, and storing caption data in which the time information is corrected. .

In this case, the modifying includes extracting only a character string from the extracted voice recognition data, extracting only a character string from the received subtitle data, aligning the character string of the voice recognition data with the character string of the subtitle data; and correcting the time information in the caption data with the extracted time information based on the alignment result.

In this case, the arranging includes calculating a similarity between the character string of the voice recognition data and the character string of the subtitle data in units of syllables, and the character string of the voice recognition data based on a syllable having a preset similarity among the calculated similarities. and aligning the character strings of the subtitle data.

In this case, in the calculating of the similarity, the similarity may be calculated based on whether the initial consonant, the middle consonant, and the final consonant are identical within a syllable unit.

In this case, the calculating of the similarity includes calculating a first value if all of the initial consonant, neutral, and final consonant within a syllable unit are the same; , a third value smaller than the second value is calculated when only one of the syllables is the same, and the step of arranging based on the syllables having the preset similarity is based on the syllables having the first value and the second value. The character string of the voice recognition data may be aligned with the character string of the subtitle data.

Meanwhile, in the arranging based on the syllables having the preset similarity, the character string of the voice recognition data and the character string of the subtitle data may be aligned using dynamic programming and the calculated similarity.

Meanwhile, the modifying may include correcting time information on the first syllable of the line of the subtitle data to time information on a character string of voice recognition data corresponding to the character string based on the alignment result.

Meanwhile, in the step of extracting only the character string from the received subtitle data, the character string excluding the text auxiliary information corresponding to the context description of the image may be extracted from the received subtitle data.

Meanwhile, the caption synchronization method may further include transmitting the modified caption data to an external device.

Meanwhile, in a computer-readable recording medium including a program for executing the subtitle synchronization method according to an embodiment of the present disclosure, the subtitle synchronization method includes: receiving image content and subtitle data for the image content; extracting voice recognition data and time information by performing voice recognition on the video content; comparing the extracted voice recognition data with the received subtitle data to correct time information in the subtitle data with the extracted time information and storing the subtitle data in which the time information is corrected.

As described above, according to various embodiments of the present disclosure, it is possible to automatically synchronize subtitle data using the voice recognition result for video content. In particular, the synchronization can be performed with high accuracy by aligning the speech recognition result and the subtitle data using the dynamic programming method.

1 is a view showing the configuration of a content creation system according to an embodiment of the present disclosure;
2 is a diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure;
3 is a diagram illustrating an example of a subtitle generated using a voice recognition method;
4 is a diagram showing an example of a closed caption;
5 is a view for explaining a dynamic programming method according to an embodiment of the present disclosure;
6 is a view for explaining a similarity calculation operation according to an embodiment of the present disclosure;
7 is a diagram comparing the alignment results when the similarity is calculated based on the identity of syllables and when the similarity is calculated by dividing the syllable into three phonemes;
8 is a diagram illustrating an example of caption data modified according to the present disclosure, and;
9 is a flowchart illustrating a caption synchronization method according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure are selected as currently widely used general terms as possible while considering the functions in the present disclosure, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

Embodiments of the present disclosure may be subjected to various transformations and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, and it should be understood to include all transformations, equivalents and substitutions included in the spirit and scope of the disclosure. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this application, "includes." Or "consistent." The term such as is intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features or number, step, operation, component, part or It should be understood that it does not preclude the possibility of the existence or addition of combinations thereof.

In an embodiment of the present disclosure, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a diagram showing the configuration of a content creation system according to an embodiment of the present disclosure.

Referring to FIG. 1 , a content generation system 1000 according to an embodiment of the present disclosure may include a broadcast transmission device 10 , a caption generation device 20 , a server 30 , and an electronic device 100 . .

The broadcast transmitting device 10 may transmit a terrestrial broadcast using image content. In this case, the broadcast transmitting device 10 may provide a caption broadcasting service by using caption data for the image content generated by the caption generating device 20 . Here, the closed caption broadcasting service is a service that displays text corresponding to a voice signal as subtitles on the TV screen. It is a technology that converts and transmits an auditory message of a TV program into an electronic code form for the hearing impaired and displays it as commentary subtitles on the TV screen. am.

In addition, the broadcast transmission device 10 may transmit the image content on which the terrestrial broadcast transmission has been completed to the server 30 . In this case, the broadcast transmission device 40 may transmit the video content to the server 30 as it is, and perform an operation of additionally inserting advertisements, etc. or dividing the video content into a plurality of pieces, that is, transmitting the edited content to the server 30 . may be

The caption generating device 20 may generate caption data for the image content transmitted from the broadcast transmitting device 40 , and transmit the generated caption data to the server 30 and/or the broadcast transmitting device 40 . Here, the caption data is data generated through a typing input of a stenographer, and may be a commentary caption transmitted in the form of an electronic code corresponding to the video content. Such subtitle data is generated by input of a professional stenographer, and has high accuracy, but takes time to input the stenography, and there is a time offset from the video content. In addition, the caption data may include text auxiliary information for describing a situation of an image for explaining a vehicle horn, a cell phone sound, etc. as well as a human voice in general. The caption data generated by such a stenographer may be referred to as a closed caption.

The server 30 may store the video content transmitted from the broadcast transmitting device 10 , and may store caption data for the corresponding video content from the caption generating device 20 . Such a server 30 may be an archive server.

The electronic device 100 may receive caption data from the server 30 and correct a sync of the received caption data. In addition, the electronic device 100 may transmit the corrected caption data to the server 30 .

In this case, the electronic device 100 receives the image content corresponding to the caption data from the server 30 , performs speech to text on the received image content, and performs text on the voice included in the image content. It is possible to generate speech recognition data having time information (or time synchronization information) of information and text, respectively. Such voice recognition data may be referred to as a voice recognition caption, a voice recognition result, and the like.

When the voice recognition data is generated, the electronic device 100 may compare the subtitle data with the extracted voice recognition data to correct time information in the subtitle data to the time information in the voice recognition data. A detailed configuration and operation of the electronic device 100 will be described later with reference to FIG. 2 .

As described above, the content generating system 1000 according to the present embodiment corrects time information in the subtitle data using voice recognition data generated through voice recognition for video content, and automatically synchronizes the subtitle data without human intervention. can be fitted to the video.

Meanwhile, in the illustration and description of FIG. 1 , each device is illustrated in a form directly connected to each other, but in implementation, each element may be connected via a separate external element. In addition, although it has been described that the electronic device 100 is connected to only one server 30 , the electronic device 100 may be connected to a plurality of servers, and may individually receive image content and subtitle data from each server. .

Also, in the illustration and description of FIG. 1 , the server 30 and the electronic device 100 are illustrated and described as separate devices, but in implementation, the two devices may be implemented as one. That is, the server 30 that stores the image content may perform a synchronization operation on the subtitle data by itself without performing subtitle synchronization by transmitting the image content to the electronic device 100 .

2 is a diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 may include a communication device 110 , a memory 120 , a display 130 , a manipulation input device 140 , and a processor 150 . Here, the electronic device 100 may be a PC capable of image processing, a notebook PC, a smart phone, a server, or the like.

The communication device 110 is connected to the caption generating device 20 and/or the server 30 , and may transmit/receive video content and caption data. Specifically, the communication device 110 is formed to connect the electronic device 100 to an external device, and is connected to a mobile device through a local area network (LAN) and the Internet network, as well as a USB (Universal) device. It is also possible to connect through a serial bus) port.

In addition, the communication device 110 may be connected to other electronic devices via a router or router connected to a public Internet network as well as a wired method, and may use a wired method with the router or router as well as Wi-Fi, Bluetooth, cellular communication, etc. It can also be connected wirelessly.

In addition, the communication device 110 may transmit subtitle data with time information corrected (ie, synchronized subtitle data) to an external device (eg, the server 30 ) by a process described later.

The memory 120 is a component for storing O/S for driving the electronic device 100 or software and data for performing voice recognition/subtitle synchronization. The memory 120 may be implemented in various forms such as RAM, ROM, flash memory, HDD, external memory, memory card, etc., but is not limited thereto.

The memory 120 stores image content. Here, the video content includes not only video but also audio, and may be a video file such as MP4, AVI, or MOV. On the other hand, the above-described file format is not an example, and is not limited to the above-described examples.

In addition, the memory 120 stores subtitle data corresponding to the above-described image content. Here, the caption data may include string information to be displayed on the caption service and time information for displaying the text string information. Here, the time information may be time information in ms units.

In addition, the memory 120 may store character string information, time information, similarity information, etc. generated during a process to be described later, and may store subtitle data in which the time information is corrected.

The display 130 displays a user interface window for receiving a selection of a function supported by the electronic device 100 . Specifically, the display 130 may display a user interface window for receiving selections of various functions provided by the electronic device 100 . The display 130 may be a monitor such as an LCD, CRT, or OLED, and may be implemented as a touch screen capable of simultaneously performing the functions of the manipulation input device 140 to be described later.

The manipulation input device 140 may receive, from a user, a function selection of the electronic device 100 and a control command for the corresponding function. Specifically, the manipulation input device 140 may receive a command for whether to correct the sync of the received caption data. In addition, the manipulation input device 140 may receive selection of subtitle data (or image content) for which subtitle synchronization is to be performed.

The processor 150 controls each component in the electronic device 100 . Specifically, when a booting command is input by the user, the processor 150 may perform booting using the operating system stored in the memory 120 .

When video content or caption data for performing caption synchronization is selected, the processor 150 may control the communication device 110 to receive caption data corresponding to the selected video content or video content corresponding to the selected caption data. .

In addition, when video content and caption data corresponding to the video content are received, the processor 150 may perform voice recognition on the video content to extract voice recognition data and time information. For example, the processor 150 may perform speech recognition using any one of widely known libraries related to speech recognition (eg, Kaldi, CMU Sphinx, HTK, AWS Transcribe).

In addition, the processor 150 may compare the extracted voice recognition data with the caption data to correct time information in the caption data to the extracted time information. Specifically, the processor 150 may extract only the character string from the previously extracted voice recognition data. For example, the voice recognition data may include a character string and time information of the corresponding character string as described above, and for a process to be described later, the processor 150 may extract only the character string from the voice recognition data.

In addition, the processor 150 may extract only the character string from the received caption data. For example, the subtitle data may include symbols such as '?' and '!', but the character string through voice recognition does not include such symbols. Accordingly, the processor 150 may extract only the character string excluding the above-described symbols from the received caption data. In addition, the caption data may include text such as information for describing a situation (text auxiliary information) (eg, a phone ringing, a car horn ringing). However, since such text auxiliary information is information that is not extracted during the speech recognition process, the processor 150 may extract a character string excluding the text auxiliary information from the received caption data. For example, in the text auxiliary information described above, there is a case where a character string is written in parentheses "(", ")". Accordingly, the processor 150 may extract a character string excluding the character string in the parenthesis column from the received subtitle data.

In addition, the processor 150 may align the character string of the voice recognition data and the character string of the subtitle data. Specifically, the processor 150 may align the character string of the voice recognition data and the character string of the subtitle data using a dynamic programming method. Here, dynamic programming is a task that aligns two long strings and is widely used to analyze genome sequences in the field of bioinformatics. Although there are many algorithms in the dynamic programming method, the Needleman-Wunsch algorithm is used in the present disclosure, but is not limited thereto. A method using the Needleman-Wunsch algorithm will be described in detail with reference to FIG. 5 .

In addition, the processor 150 may correct the time information in the caption data to the extracted time information based on the alignment result. Specifically, the processor 150 may correct the time information about the starting point (ie, the first syllable) of the line of the subtitle data to the time information about the character string of the voice recognition data corresponding to the corresponding character string based on the alignment result. For example, as shown in FIG. 4 , 28025 time information (ie, 28.025 seconds) for the string 'Chief General~' is time information corresponding to 'Chief Manager' in the voice recognition result (for example, .

In addition, the processor 150 may store the subtitle data in which the time information is corrected in the memory 120 . Meanwhile, when the caption data is stored in an external device (eg, the server 30) and provided to other devices, the processor 150 controls the communication device 110 to transmit the corrected caption data to the external device. can do.

Meanwhile, in the above description, the electronic device 100 modifies (or updates) the time information of the caption data, but the above-described operation generates new caption data by combining the string information of the caption data and the time information of the voice recognition data. It can also be expressed as

As described above, the electronic device 100 according to the present embodiment corrects time information in the caption data by using the speech recognition data generated through speech recognition for video content, so that it is easier and faster to synchronize the caption data to the video data. it is possible

On the other hand, in the illustration and description of FIGS. 1 and 2, it has been illustrated and described that the above-described operation is performed only for content generated by a broadcasting station, but separate subtitle data input by the stenographer for video content generated for other than the broadcasting station If there is, it can be applied.

Meanwhile, in the illustration and description of FIG. 2 , the electronic device 100 is illustrated and described as including a display and a manipulation input device. However, when the electronic device 100 is implemented as a device such as a server, the display and The manipulation input device may be omitted.

FIG. 3 is an example of a caption generated using a voice recognition method for video data, and FIG. 4 is a diagram illustrating an example of a closed caption for the video data of FIG. 3 .

Referring to FIG. 3 , speech recognition technology for video content has significantly evolved due to the development of speech recognition technology. Since voice recognition is performed on video content, the time synchronization information for the calculated text is very accurate. However, as shown, the text, which is the recognition result, has a different part from the actual user's utterance. In particular, comparing with the closed caption of FIG. 4 directly input by the stenographer for this, it can be confirmed that the caption directly input by the stenographer is excellent in terms of accuracy.

Referring to FIG. 4 , caption data includes a text string (ie, text information) for a caption and time information (or time synchronization information) at which the text string is to be displayed in an image. The time information is displayed in the form of <SYNC Start=...> as shown. In the example shown, the word 'General Manager' at the beginning starts at 28.025 seconds. However, referring to FIG. 3 , it can be seen that the portion misrecognized as 'same-same as' actually starts at 18.72 seconds in the image. That is, it can be seen that the closed caption has an offset of approximately 10 seconds from the actual image.

Although the closed caption input by the stenographer is highly accurate in terms of content, an offset occurs due to the process of storing the closed caption, broadcast production and archiving process, etc., so the time synchronization information in the closed caption is different from the actual video. Conversely, the voice recognition subtitles obtained by performing voice recognition have relatively accurate time synchronization information, but the content of the text contains many errors. Advantages and disadvantages of such speech recognition captions and closed captions can be summarized in Table 1 below.

text information time synchronization information closed captions exact inaccuracy Voice Recognition Subtitles inaccuracy exact

Accordingly, in the present disclosure, caption data having high accuracy in both text information and time synchronization information can be generated by using text information of closed captions, but synchronizing time synchronization information with time information of voice recognition captions.

Meanwhile, in order to compare the text of the closed caption with the text of the speech recognition, the present disclosure uses dynamic programming. Numerous texts exist within closed captions, and the same phrase may be included repeatedly. Therefore, when arranging by simply comparing the positions of specific words, the two texts may not be aligned. In this regard, the present disclosure aligns the two texts using dynamic programming, which is widely used for sequencing. A specific operation will be described below with reference to FIG. 5 .

5 is a diagram for explaining a dynamic programming method according to an embodiment of the present disclosure.

Dynamic programming is a method of finding the answer to the whole problem based on the answers to the subproblems in each step when a problem consists of repeated subproblems in multiple steps. This dynamic programming method is a method of solving problems from the smallest subproblem to the higher level problem.

Referring to FIG. 5 , the upper part of the table is base information constituting the first base sequence, and the left side of the table is base information constituting the second base sequence. When the comparison operation using the dynamic programming method is performed, the similarity of the two base information is compared from the bottom right of the table, and finally, the comparison is made up to the top left of the table. Through this process, the entire alignment of the two nucleotide sequences can be performed.

Meanwhile, in order to apply this dynamic programming method to a character string, the similarity may be calculated by taking the smallest partial problem as a syllable, that is, in units of syllables. A detailed similarity calculation operation will be described later with reference to FIG. 6 .

In addition, when a path having the maximum cumulative similarity value as in FIG. 5 is found using the calculated similarity, the two strings may be finally sorted.

6 is a diagram for explaining a similarity calculation operation according to an embodiment of the present disclosure.

Each score for setting the similarity matrix in the Needleman-Wunsch algorithm can be set as follows.

match_score = similarity(syllable 1, syllable 2)

gap_score = -1

Specifically, by comparing two syllables, if they are the same, the degree of similarity may be calculated as a first value (+1), and if they are not identical or there is information omission, it may be calculated as a second value (-1).

However, in the case of performing in the above-described manner, if the syllable shape is different even if the pronunciation is similar, such as 'gong' - 'bong', the above method is judged as a mismatch, and thus performance degradation may occur.

Therefore, after decomposing a syllable into phonemes of the unit of beginning/middle/final consonants, the similarity may be calculated by comparing each phoneme in units of beginning/middle/final consonants. For example, comparing the leading/middle/final voice of each syllable, if all phonemes are the same, the first value (eg 3), if one of the leading/neutral/final voices is different, the second value (eg 2), If only one of /neutral/finality is the same, it can be calculated as a third value (eg, 1).

Examples of comparing two syllables through this process are shown in FIG. 6 . For example, if the initial, middle, and final phonemes of two syllables are the same, such as 'ga' and 'ga', 3 points can be calculated as similarities. A point may be calculated, and if only the initial consonant is the same, such as 'yo' or ;won, one point of similarity may be calculated.

Meanwhile, in the above, only the method of assigning +3/+2/+1 points in units of phonemes has been described. However, in implementation, the above-described values cannot be illustrated, and a value different from the numerical value above may be used, and seconds/ A similarity value may be calculated by assigning different weight values to the middle/last grade. For example, if three phonemes are the same, +5 points, if only some phonemes are the same, if the leading consonants match, +2 points, and neutral/final consonants +1 point.

7 is a diagram comparing the alignment results between the case where the degree of similarity is calculated based on whether syllables are identical and the case where the degree of similarity is calculated by dividing the syllable into three phonemes.

Specifically, the left side of the table is the result of calculating the similarity based only on whether syllables are identical, and the right side of the table is the alignment result when the similarity is calculated by dividing the syllable into three phonemes.

Referring to FIG. 7 , it can be confirmed that more accurate alignment is possible when the similarity is calculated by dividing a syllable into three phonemes.

8 is a diagram illustrating an example of caption data modified according to the present disclosure.

The result of FIG. 8 is calculated by performing subtitle synchronization according to the present disclosure on video content of approximately 28 minutes and 24 seconds. In the case of text accuracy, as described above, closed captions are used as they are, so the accuracy is high. And the average synchronization error per syllable of the subtitles was about 0.19 syllables. From this result, it can be seen that most of the subtitles are accurately aligned in units of syllables.

Meanwhile, the above-described subtitle synchronization operation took a total of 4 minutes and 7 seconds, and the process of aligning the voice recognition subtitles and closed subtitles and correcting the time information took 78 seconds among the entire process. That is, during the entire time operation, the speech recognition process takes time. This is because the cloud system was used for voice recognition in the above-described test process. If the voice recognition operation is performed by itself, the time required for overall synchronization can be further reduced.

Meanwhile, in the above, only the synchronization operation for the Korean subtitles has been described, but when implemented, it can be applied to languages other than Korean (eg, English, Chinese, Japanese, etc.). For example, when applied to English subtitles, two English texts may be compared by using the above-mentioned phoneme as an alphabet and using several alphabet combinations (ie, vowel + consonant) as the above-mentioned syllable. In addition, when applied to Chinese subtitles, two Chinese texts may be compared by using Pinyin corresponding to one Chinese character as the above-mentioned syllable, and using the sungmo/mica constituting one Pinyin as the above-mentioned phoneme.

9 is a flowchart illustrating a caption synchronization method according to an embodiment of the present disclosure.

Referring to FIG. 9 , first, image content and caption data for the image content are received ( S910 ). For example, when video content or caption data to be synchronized with caption is selected, the selected video content or caption data may be received, and caption data or video content corresponding to the received video content or received caption data may be searched and received. there is. Meanwhile, in implementation, at least one of image content or caption data may be pre-stored, and in this case, the above-described receiving operation may be omitted.

Then, voice recognition is performed on the video content to extract voice recognition data and time information (S920). Specifically, by using the voice recognition library to recognize the voice of the video content, it is possible to generate voice recognition data having text for the voice and time information of the text.

Then, by comparing the extracted voice recognition data with the received caption data, time information in the caption data is corrected to the extracted time information (S930). For example, first, only the character string may be extracted from the extracted voice recognition data, and only the character string may be extracted from the received subtitle data. In addition, the character string of the voice recognition data and the character string of the caption data may be aligned, and time information in the caption data may be corrected to the extracted time information based on the alignment result. Since each specific operation has been described above, a redundant description thereof will be omitted.

The subtitle data in which the time information is corrected is stored (S940). If the subtitle data is received from the external device, the subtitle data in which the time information is corrected may be transmitted to the external device.

Accordingly, in the file transmission method according to the present embodiment, time information in the subtitle data is corrected using voice recognition data generated through voice recognition of video content, so that the synchronization of the subtitle data can be easily matched with the image. The caption synchronization method shown in FIG. 10 may be executed on the electronic device having the configuration of FIG. 2 and may also be executed on the electronic device having other configurations.

In addition, the subtitle synchronization method as described above may be implemented as a program including an executable algorithm that can be executed on a computer, and the above-described program is stored in a non-transitory computer readable medium to be provided. can

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device. Specifically, the programs for performing the above-described various methods may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure belongs without departing from the gist of the present disclosure as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present disclosure.

1000: content creation system 10: broadcast transmission device
20: subtitle generating device 30: server
100: electronic device 110: communication device
120: memory 130: display
140: operation input device 150: processor

Claims (20)

In an electronic device,
a communication device for receiving video content and caption data for the video content;
Memory; and
performing voice recognition on the video content to extract voice recognition data and time information, comparing the extracted voice recognition data with the received caption data, and correcting time information in the caption data with the extracted time information; , a processor that stores the subtitle data of which time information is corrected in the memory;
The processor is
Extracting only a character string from the extracted voice recognition data, extracting only a character string from the received subtitle data, and extracting the character string of the voice recognition data and the character string of the subtitle data within a syllable unit, The similarity is calculated by giving a weight value, and the character string of the voice recognition data and the character string of the subtitle data are sorted based on a syllable having a preset similarity among the calculated similarities, and the character string of the subtitle data is selected based on the alignment result. An electronic device for correcting time information with the extracted time information. delete delete delete delete delete According to claim 1,
The processor is
The electronic device for correcting the time information of the first syllable of the line of the subtitle data to the time information of the character string of the voice recognition data corresponding to the character string based on the alignment result. According to claim 1,
The processor is
An electronic device for extracting a character string from the received caption data excluding text auxiliary information corresponding to a context description of an image. According to claim 1,
The caption data for the video content includes:
An electronic device that is a caption for commentary transmitted in the form of an electronic code in response to the video content. delete A method for synchronizing subtitles in an electronic device, the method comprising:
receiving video content and caption data for the video content;
performing voice recognition on the video content to extract voice recognition data and time information;
comparing the extracted speech recognition data with the received caption data and correcting time information in the caption data with the extracted time information; and
Storing subtitle data in which time information is corrected;
The modifying step is
extracting only a character string from the extracted voice recognition data;
extracting only a character string from the received caption data;
The character string of the speech recognition data and the character string of the subtitle data are given different weight values for the initial consonant, middle, and final consonant within a syllable unit to calculate the similarity, and based on the syllable having a preset similarity among the calculated similarities aligning the character string of the speech recognition data and the character string of the subtitle data; and
and modifying time information in the caption data to the extracted time information based on the alignment result. delete delete delete delete delete 12. The method of claim 11,
The step of correcting the time information in the caption data to the extracted time information based on the alignment result includes:
A subtitle synchronization method for correcting time information of a first syllable of a line of the subtitle data to time information of a character string of speech recognition data corresponding to the character string based on the alignment result. 12. The method of claim 11,
The step of extracting only the character string from the received subtitle data includes:
A caption synchronization method for extracting a character string from the received caption data excluding text auxiliary information corresponding to a contextual description of an image. delete A computer-readable recording medium comprising a program for executing a subtitle synchronization method,
The subtitle synchronization method includes:
receiving video content and caption data for the video content;
performing voice recognition on the video content to extract voice recognition data and time information;
comparing the extracted speech recognition data with the received caption data and correcting time information in the caption data with the extracted time information; and
Storing the subtitle data in which the time information is corrected;
The modifying step is
extracting only a character string from the extracted voice recognition data;
extracting only a character string from the received caption data;
The character string of the speech recognition data and the character string of the subtitle data are given different weight values for the initial consonant, middle, and final consonant within a syllable unit to calculate the similarity, and based on the syllable having a preset similarity among the calculated similarities aligning the character string of the speech recognition data and the character string of the subtitle data; and
and modifying the time information in the subtitle data to the extracted time information based on the alignment result.

Images