KR102463283B1 - automatic translation system of video contents for hearing-impaired and non-disabled - Google Patents

Abstract

The present invention relates to an automatic translation system of video content for both the hearing-impaired and the non-disabled and, more specifically, to an automatic translation system of video content for both the hearing-impaired and the non-disabled, which outputs a sign language to enable the hearing-impaired to enjoy video content and the non-disabled to readily learn the sign language and minimizes mismatch between a mouth shape and a voice on the video when the video is dubbed with the other language to allow a viewer to enjoy the video in a natural way. The automatic translation system of the present invention is used to perform: a voice extraction step of extracting voice information from the video content; a voice recognition step of converting the voice extracted in the voice extraction step into an original language text; a translation step of translating the original language text extracted in the voice recognition step into a selected language; a voice conversion step of converting the text translated in the translation step into a voice; and a dubbing step of combining the voice information derived in the voice conversion step with the original video content.

Description

Automatic translation system of video contents for hearing-impaired and non-disabled

The present invention relates to a system for automatic translation of video content for both the hearing impaired and the non-disabled, and more specifically, by outputting sign language, not only the hearing impaired can enjoy the video content, but also the general public can more easily learn sign language, The present invention relates to an automatic translation system for video content for the hearing impaired and non-disabled that enables users to enjoy video naturally by minimizing the mismatch between the mouth shape and the voice of the video even when dubbing in another language.

Recently, media services that handle user-generated content such as YouTube and TikTok are rapidly expanding.

Most of the content provided by these media services is video, and research and interest in technology for providing audio signals included in video as subtitles are also increasing significantly.

In general, the caption of the video is obtained based on the result of a speech recognition (STT: Speech To Text) for the voice signal of the video or the operator listens to the voice of the video. In the former case, very much time, money, and effort are required. Also, in both cases, it is difficult to display the subtitles acquired during video playback in synchronization with the audio signal, and there is a limitation in that subtitles can only be provided with a uniform length depending on the media service.

On the other hand, due to the development of the Internet, a video provided by a media service is widely played by global users who speak a language other than the language in which it was produced. Although translated subtitles are provided in some media services, the level has not reached the level that satisfies consumers.

So, in order to solve this problem, the technology described in Korean Patent Registration No. 10-2351400 (registered on November 11, 2022) as shown in FIGS. 1 and 2 has been proposed, and its technical characteristics are the generating a first time table by obtaining a subtitle sentence from a voice signal, and matching start time information and end time information of a corresponding voice signal with respect to each word of the subtitle sentence; Based on the subtitle display unit set as a multiple of the character unit of the first language, the subtitle sentence is divided into a plurality of subtitle groups grouped by word unit of the first language, and the plurality of subtitle groups are generating a second timetable including start and end time information for each of the subtitle groups; and a first area for synchronizing the plurality of caption groups with the audio signal and displaying them together with the moving picture, a second area for providing a list of the plurality of caption groups, and a third area for setting caption related matters. It characterized in that it comprises the step of providing to the user terminal.

However, the technology described in Korean Patent No. 10-2351400 (registered on 11/11/2022) has the advantage of extracting voice information from a video, translating it into a specific language, and synchronizing it with a voice signal, but When watching a video, obtaining information through subtitle information has a problem in that the viewer cannot concentrate on the video.

In addition, there is a problem in that a considerable cost is required to produce a video dubbed in the viewer's language.

Korean Patent No. 10-2351400 (Registered on 11/11/2022)

The present invention has been devised to solve the above problems, and an object of the present invention is not only to help language learning by adding original text or translated text to video content, but also to convert the original text and the translated text into the original sign language and the number of translations, respectively. It is to provide an automatic translation system for video content for the hearing impaired and non-disabled that allows the hearing impaired to enjoy the video content and the general public can more easily learn sign language by converting it into a language and adding it.

Another object of the present invention is to extract text information from a voice signal of video content, translate the extracted text information into a desired language, and then automatically dub it using the voice stored in the voice database, thereby reducing the cost. It is to provide an automatic translation system for video content for the hearing impaired and non-disabled that allows users to watch video content in a specific language in their native language or in their preferred language.

In addition, another object of the present invention is that, when dubbing video content in a specific language into another language, the difference in length between the language of the original and the language of the translated version is made within an error during the translation process so that the output voice is It is to provide an automatic translation system for video content for the hearing impaired and the non-disabled that allows the length to correspond to the image of the video to enjoy a natural dubbing video.

The present invention for solving these problems;

A voice extraction step of extracting voice information from the video content, a voice recognition step of converting the voice extracted in the voice extraction step into original text and converting the original text into a sign language, and the original text information extracted in the voice recognition step is translated into the selected language and characterized in that it comprises a translation step of converting the translated text into a sign language of the selected language.

Here, the step of converting the text translated in the translation step into speech, and a dubbing step of merging the audio information derived in the speech conversion step into the original video content are characterized in that it further comprises.

Meanwhile, in the speech recognition step, the starting and ending points of each sentence are checked, and the entire syllable of the corresponding sentence is derived.

In this case, in the voice recognition step, it is characterized in that the number of characters appearing in the video content is checked.

And, in the translation step, a plurality of translations are generated with various expressions, and the syllables of the plurality of translations are compared with the syllables of the original sentence to select a translation having a difference in the number of syllables within a predetermined range.

Here, in the speech conversion step, the output time of the original sentence is checked through the start and end points of each sentence, and the speed of the voice is adjusted so that the output time of the converted voice information is the same as the output time of the original sentence. characterized.

In this case, the voice conversion step is characterized in that the voice information stored in the voice database is matched with voices having different characteristics according to each character.

On the other hand, the dubbing step deletes the original language audio information from the video content, combines the translated audio information generated in the audio conversion step with the video information so that the starting and ending points match, and the original text and the translated text are added to the video information. It is characterized by further including text, original sign language, and translated sign language information.

Here, a learning step for learning a foreign language or sign language is further performed after the dubbing step, and the learning step is characterized in that it consists of a section repetition learning process or a comparative learning process selected and performed by a user.

In this case, the section iterative learning process repeatedly outputs a section selected by the user in the video content derived through the dubbing step, and any one of original text, translated text, original sign language, and translated sign language information according to user settings It is characterized in that it outputs an abnormality.

In addition, the comparative learning process includes a sign language detection process for converting the sign language detected in the user's sign language reproduction image into original text, a translation process for translating the original text into text of the selected language, and converting the original text into a standard original sign language It is characterized in that it consists of a standard sign language conversion process and a comparative output process of simultaneously outputting the original text, the translated text, and the standard original sign language to the user's sign language reproduction image.

According to the present invention of the above configuration, by adding the original text or translated text to the video content to help language learning, and by converting the original text and the translated text into the original sign language and the translated sign language, respectively, and adding it, the hearing impaired can also You can enjoy video content, and it has the effect of making it easier for the general public to learn sign language.

In addition, the present invention extracts text information from a voice signal of video content, translates the extracted text information into a desired language, and then automatically dubs it using the voice stored in the voice database, so that it can be translated into a specific language at a low cost. It has the effect of allowing you to watch the video content in your native language or in your preferred language.

In addition, in the present invention, when dubbing video content made in a specific language into another language, the length difference between the original language and the translated version is made within an error during the translation process, so that the length of the output voice is the same as that of the image even after dubbing. It has the effect of allowing you to enjoy a natural dubbing video by making it correspond to the image.

1 is a conceptual diagram of a conventional video translation subtitle providing system.
2 is a flowchart of a conventional video translation subtitle providing system.
3 is a conceptual diagram of a system for automatic translation of video content for both the hearing impaired and the non-disabled according to the present invention.
4 is a block diagram of a system for automatic translation of video content for both the hearing impaired and the non-disabled according to the present invention.
5 is a flowchart of a system for automatic translation of video content for both the hearing impaired and the non-disabled according to the present invention.
6 is a flowchart of a system for automatic translation of video content for the hearing impaired and the non-disabled according to another embodiment of the present invention.
7 is a block diagram of a system for automatic translation of video content for the hearing impaired and the non-disabled according to the embodiment of FIG. 6 .

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted. And, it should be understood that the present invention may be implemented in many different forms and is not limited to the described embodiments.

3 is a conceptual diagram of a system for automatic translation of video content for the hearing impaired and non-disabled according to the present invention, FIG. 4 is a block diagram of a system for automatic translation of video content for the hearing impaired and non-disabled according to the present invention, and FIG. 5 is a diagram of the present invention 6 is a flowchart of a system for automatic translation of video content for the hearing impaired and non-disabled according to another embodiment of the present invention, and FIG. It is a block diagram of a system for automatic translation of video content for both the hearing impaired and the non-disabled according to an example.

The present invention relates to a system for automatic translation of video content for the hearing impaired and the non-disabled. As shown in FIGS. 3 to 5, the configuration is located at a remote location from the user terminal 200 connected through a network network, and the user terminal ( 200) and the server 100 connected by communication.

Here, if the user wants to watch video content in his or her mother tongue because it is in a foreign language or wants to watch video content in a specific foreign language despite being in his or her mother tongue, upload the video content to the server 100 through the user terminal 200 and A desired language is dubbed into the corresponding image content through the automatic translation system for the deaf and non-disabled users of the present invention provided in the server 100 .

Therefore, the image content uploaded by the user through the user terminal 200 is dubbed in the user's selected language and transmitted to the user terminal 200, and the user can enjoy the dubbed image content in the desired language.

And, as shown in FIG. 5, the automatic video content translation system of the present invention converts the audio extracted in the audio extraction step (S100) of extracting audio information from the video content and the audio extraction step (S100) into original text. A translation step (S300) of translating the original text information extracted in the speech recognition step (S200) and the speech recognition step (S200) into a selected language, and a speech conversion of converting the text translated in the translation step (S300) into speech A dubbing step (S500) of combining the audio information derived in the step (S400) and the audio conversion step (S400) with the original video content is performed.

Here, the server 100 is provided with a communication module 170 for communication with a user terminal 200 such as a smartphone possessed by the user to receive image content from the user terminal 200 .

At this time, in the audio extraction step ( S100 ), the received image content is analyzed through the analysis module 110 provided in the server 100 , and image information and sound information are separated from the image content. Background sound, sound effect, and voice are further separated from the information.

On the other hand, the server 100 is provided with a storage unit 180 to store image information, background sound, sound effect, and voice separated through the analysis module 110, respectively. In the voice recognition step (S200), the server 100 ), the separated voice information is analyzed and converted into text information through the voice recognition module 120 provided in the .

Here, in the speech recognition step (S200), the start and end points of each sentence are checked on the image information, and the total number of syllables of the corresponding sentence is derived. The voice information converted into text may be a single sentence. Also, a plurality of sentences may be connected, and the number of derived syllables is derived from the number of syllables for the entire sentence.

At this time, in the voice recognition step (S200), voice information may be analyzed through the voice recognition module 120 to derive a voice fingerprint (voice print) for each sentence, and video content through the voice fingerprint derived in this way You can check the number of characters appearing in .

Here, the voices for which it is difficult to determine whether the voice fingerprint is of the same character can be determined by comparing the image information such as the appearance and mouth shape of the character at the time of the voice utterance of a specific character, or by manually detecting whether the character is the same may input or modify specific character information corresponding to a voice at a specific location.

And, in the translation step (S300), the original text is translated into a specific language selected by the user using the translation module 130 provided in the server 100. When the translation module 130 translates the original text , instead of a single sentence, various sentences with the same meaning are generated.

Here, in the translation step (S300), the number of syllables of each of the plurality of sentences generated by the translation module 130 is derived, and the translation module 130 compares the syllables of the plurality of translations with the syllables of the original sentence. The translation with the most similar number of syllables is selected.

At this time, the translation module 130 sets an error range between the number of syllables in the original sentence and the number of syllables in the translated sentence to select only sentences within the error, thereby generating the translated voice information in the voice conversion step (S400). In this case, it is generated to have the same length as the audio information in the original language, so that the image information and the audio information do not synchronize with each other to prevent a sense of difference.

As an example of selecting the translation sentence, the line from the movie "I'm a pawnshop. I'll take the gold teeth. I'll chew everything except the gold teeth" is a sentence with 26 syllables.

If this sentence is translated into English,

The first can be translated as "I am managing a pawn shop. I'll take the gold teeth. I'll chew everything except you golden teeth", in this case the pronunciation is "I am managing a pawn shop are take the gold." It becomes 35 syllables as "Teat Isle Chew Everything Exception Your Golden Teate".

The second one can be translated as "I run a pawn shop. I'll chew and eat everything except the golden teeth", in this case the pronunciation is "I run a pawn shop. I'll chew and eat everything except the golden teeth" ' to make 24 syllables.

Accordingly, the translation module 130 selects the second translation sentence, which is 24 syllables most similar to 26 syllables, which are syllables of the original sentence, as the final translation sentence.

In addition, the translation step (S300) may further include an editing function for manually writing a translation. After automatically performing the translation step (S300), if there is no separate input for a set time, The translation is automatically written and selected through the module 130, and when an input set for performing the editing function is detected, the editing function is performed.

Here, in the case of performing the editing function, among the plurality of translations translated by the translation module 130, in the case of a translation in which the error with the number of syllables in the original text is out of a set range, it is highlighted in red, etc. so that the operator can When you manually write a translation, you increase the efficiency of your work.

On the other hand, in the speech conversion step (S400), the translated text is converted into speech using the speech conversion module 140 provided in the server 100. When converting to speech, the speech information converted into the set language is The output time is adjusted to be the same as the output time of the original speech information.

That is, it is possible to check the output time of the original sentence through the information on the start and end points of each sentence confirmed in the speech recognition step (S200). Considering the number of syllables of the translated text checked through the translation module 130 to control the speed of the output voice (the speed of speaking in the translated language).

Therefore, as described above, when viewing video content, the time of the character speaking in the video and the time of the output audio information are made to be the same so that the immersion feeling is not broken when the user enjoys the video content.

At this time, when each sentence is converted into speech in the speech conversion step ( S400 ), a specific speech is selected and used from speech information stored in the speech database 160 provided in the server 100 , as described above. Voices having different characteristics are selected as many as the number of characters derived in the voice recognition step S200 to match different voices according to each character.

In addition, when the voice recognition module 120 converts the voice into text, the voice information is analyzed and the strength, accent, high and low, intonation, duration, pause, boundary, average pitch, pitch of phonemes in the corresponding sentence are analyzed. Characteristics such as curve, utterance rate, and utterance type are extracted.

Here, in the speech conversion step (S400), the speech characteristics in each sentence extracted by the speech recognition module 120 are reflected and converted into speech through the speech conversion module 140, so that emotions included in each sentence, etc. to make it more vivid.

In the dubbing step (S500), the original speech information is deleted from the background sound, effect sound, and voice, which are sound information separated from the image information through the analysis module 110, and the translated speech information generated in the speech conversion step (S400) is combined with the video information.

Here, in the dubbing step (S500), the above-described dubbing process is performed using the dubbing module 150 provided in the server 100. When the dubbing module 150 mixes the translated voice information with the image information, , is mixed with the video information to match the starting and ending points of the original voice information confirmed in the voice recognition step ( S200 ).

Therefore, the original video content is converted into video content dubbed in a language desired by the user by finally dubbing the translated audio information into the video information.

On the other hand, in another embodiment of the present invention, as shown in FIGS. 6 and 7 , the audio extracted in the audio extraction step ( S100 ) of extracting audio information from the video content and the audio extracted in the audio extraction step ( S100 ) is converted to original text. A speech recognition step (S200) of converting and a translation step (S300) of translating the original text information extracted in the speech recognition step (S200) into a selected language, and converting the text translated in the translation step (S300) into speech It consists of a dubbing step (S500) and a learning step (S600) of combining the audio information derived in the audio conversion step (S400) and the audio conversion step (S400) with the original video content.

Here, in the speech recognition step (S200), a sign language generation process (S210) of converting the derived original text into a sign language of the original language through the sign language module 190 provided in the server 100 is further performed, and the translation In step S300, a sign language translation process (S310) of converting the translated text into a sign language of the selected language through the sign language module 190 provided in the server 100 is further performed.

Since the technology for converting sign language into text or text into sign language is already known, a separate description thereof will be omitted.

In addition, the dubbing step (S500) deletes the original audio information from the video content, combines the translated audio information generated in the audio conversion step (S400) with the video information so that the start and end points match, and the video information The original text, translated text, original sign language, and translated sign language information are further included.

Therefore, when the user outputs the dubbed video content received from the server 100 through the user terminal 200, the user outputs at least one of the original text, the translated text, the original sign language, and the translated sign language in various ways. to enjoy video content.

And, as described above, after the dubbing step (S500), a learning step (S600) for learning a foreign language or sign language is further performed, and the learning step (S600) is a section repetition learning process (S610) and a comparative learning process ( S620), and the user selects and performs either the section iterative learning process (S610) and the comparative learning process (S620).

Here, the section repetition learning process (S610) repeatedly outputs the section selected by the user from the video content derived through the dubbing step (S500), and according to the user's settings, the original text, the translated text, the original sign language and the translation Output any one or more of the signed sign language information.

In this case, the section repetition learning process ( S610 ) may be performed while interworking with the server 100 , but as described above, the section selected by the user in the process of playing the dubbed video content in the user terminal 200 . to play repeatedly for the selected number of times.

Thus, the user can easily learn a foreign language, a sign language in the original language, and a translated sign language.

In addition, although the section iterative learning process (S610) can be performed only with the user terminal 200 in a state that is not interlocked with the server 100, the comparative learning process (S620) is performed between the user terminal 200 and the server 100 ) can be performed only in the interlocked state.

Here, the comparison learning process ( S620 ) includes a sign language detection process ( S621 ) of converting a sign language detected in the user's sign language reproduction image into original text, a translation process ( S622 ) of translating the original text into text of the selected language, and the original text text It consists of a standard sign language conversion process (S623) of converting , into a standard original sign language, and a comparison output process (S624) of simultaneously outputting the original text, the translated text, and the standard original sign language to the user's sign language reproduction image.

At this time, the sign language reproduction image in which the user reproduces the sign language may be captured in real time through a camera (not shown) provided in the user terminal 200 and transmitted to the server 100 , or a pre-recorded image may be transmitted to the server 100 .

And, in the sign language detection process (S621), the sign language reproduction image received through the sign language module 190 is analyzed and converted into original text, and the standard sign language conversion process (S623) is also the original language through the sign language module 190. The text will be converted to the sign language of the selected language.

Here, since the translation process (S622) proceeds in the same manner as the above-described translation step (S300), a separate description will be omitted.

In this case, the comparison output process (S624) is performed by merging the original text, the translated text, and the standard original sign language derived through the above-described process with the sign language reproduction image through the dubbing module 150, and the user terminal ( 200), and is played through a display (not shown) of the user terminal 200 .

Therefore, the user can compare the sign language reproduced by the user with the standard original sign language, so that there is no awkwardness in the sign language implementation, so that the user can correct the sign language more accurately.

Although preferred embodiments of the present invention have been described above, the scope of the present invention is not limited thereto, and the scope of the present invention extends to those substantially equivalent to the embodiments of the present invention. Various modifications are possible by those of ordinary skill in the art to which the invention pertains without departing from the scope of the invention.

The present invention relates to a system for automatic translation of video content for both the hearing impaired and the non-disabled, and more specifically, by outputting sign language, not only the hearing impaired can enjoy the video content, but also the general public can more easily learn sign language, The present invention relates to an automatic translation system for video content for the hearing impaired and non-disabled that enables users to enjoy video naturally by minimizing the mismatch between the mouth shape and the voice of the video even when dubbing in another language.

100: server 110: analysis module
120: speech recognition module 130: translation module
140: voice conversion module 150: dubbing module
160: voice database 170: communication module
180: storage unit 200: user terminal

Claims (11)

An audio extraction step of extracting audio information from video content;
A voice recognition step of converting the voice extracted in the voice extraction step into original text and converting the original text into sign language;
a translation step of translating the original text information extracted in the speech recognition step into a selected language and converting the translated text into a sign language of the selected language,
A voice conversion step of converting the text translated in the translation step into speech, and a dubbing step of merging the audio information derived in the speech conversion step with the original video content,
In the speech recognition step, the starting and ending points of each sentence are identified, and the entire syllable of the corresponding sentence is derived,
In the translation step, a plurality of translations are generated with various expressions,
A system for automatic translation of video content for the hearing impaired and non-disabled, characterized in that by comparing the syllables of a plurality of translations with the syllables of the original sentence, a translation having a difference in the number of syllables is selected within a predetermined range.
delete delete According to claim 1,
In the voice recognition step, the automatic translation system for video content for the hearing impaired and non-disabled, characterized in that the number of characters appearing in the video content is checked.
delete 5. The method of claim 4,
In the speech conversion step, the output time of the original sentence is checked through the starting and ending points of each sentence,
A combined video content automatic translation system for the hearing impaired and non-disabled, characterized in that the audio speed is adjusted so that the output time of the converted voice information is the same as the output time of the original sentence.
7. The method of claim 6,
In the voice conversion step, the automatic translation system for video content for the hearing impaired and non-disabled, characterized in that the voices of different characteristics according to each character are matched in the voice information stored in the voice database.
5. The method of claim 4,
The dubbing step deletes the original language audio information from the video content,
The translated voice information generated in the voice conversion step is combined with the video information so that the starting and ending points are correct,
The system for automatic translation of video content for the hearing impaired and non-disabled, characterized in that the image information further includes the original text, the translated text, the original sign language, and the translated sign language information.
9. The method of claim 8,
A learning step for learning a foreign language or sign language is further performed after the dubbing step,
The learning step is an automatic translation system for video content for the hearing impaired and non-disabled, characterized in that it consists of a section repeat learning process or a comparative learning process that is selected and performed by the user.
10. The method of claim 9,
The section iterative learning process repeatedly outputs the section selected by the user from the video content derived through the dubbing step,
A combined image content automatic translation system for the hearing impaired and non-disabled, characterized in that one or more of the original text, the translated text, the original sign language, and the translated sign language information are output according to the user's setting.
10. The method of claim 9,
The comparative learning process includes a sign language detection process of converting the sign language detected in the user's sign language reproduction image into original text;
a translation process of translating the original text into text in the selected language;
A standard sign language conversion process for converting the original text into a standard original sign language;
A combined image content automatic translation system for the hearing impaired and non-disabled, characterized in that it comprises a comparative output process of simultaneously outputting the original text, the translated text, and the standard original sign language to the sign language reproduction image of the user.

Images