WO2018121001A1

WO2018121001A1 - Method and system for outputting simultaneous interpretation of digital television program, and smart terminal

Info

Publication number: WO2018121001A1
Application number: PCT/CN2017/106377
Authority: WO
Inventors: 何加军
Original assignee: 深圳市九洲电器有限公司
Priority date: 2016-12-30
Filing date: 2017-10-16
Publication date: 2018-07-05
Also published as: CN106791913A

Abstract

The present application relates to a method and system for outputting simultaneous interpretation of a digital television program, and a smart terminal. The method comprises: separating the video, audio, and subtitle of a cached television program data stream, performing segmentation, timbre recognition, translation, and other processing on audio data, performing correction and synchronization by using subtitle data and timestamps to simultaneously interpret the original audio data into audio data in a language that is required by a user, and playing the audio for the user, so that the user is capable of understanding the audio of a television program without watching subtitles, thereby bringing great convenience to users when watching television programs, preventing the users from missing frame contents of the television programs, and greatly improving watching experience of the users.

Description

Simultaneous translation output method, system and intelligent terminal for digital television program

Technical field

The present application relates to the field of digital television, and in particular, to a method and system for simultaneous interpretation of digital television programs and an intelligent terminal.

Background technique

At present, the digital television program sounds played by the digital TV set-top box (or television) are the original sounds corresponding to the programs, so that the user can watch the television programs in an original manner.

However, the original sound of the program may be a foreign language, such as an English TV program. TV programs that are spoken in a foreign language often provide bilingual subtitles for viewers who do not understand foreign languages. If the audience does not understand foreign languages, they can only rely on the Chinese subtitles at the bottom of the screen to understand the content of the TV programs. The Chinese subtitles below often do not take into account the content of the TV program, which will greatly affect the viewer's viewing effect, so that the viewer can not watch the TV program very well, causing inconvenience to the audience.

Application content

In view of this, it is necessary to provide a method, system and intelligent terminal for simultaneous interpretation of digital television programs in view of the above-mentioned foreign language sounding television programs, viewers watching Chinese subtitles affecting viewers watching television programs, causing inconvenience.

A method for simultaneous interpretation output of a digital television program provided by an embodiment of the present application includes the following steps:

S10: controlling the audio and video terminal to buffer the stored television program data stream;

S20: parsing and separating the video data, the audio data, and the subtitle data from the buffered television program data stream, and marking the time stamp when separating, marking the synchronization label for the three;

S30: segmenting the audio data, and decoding the segmented audio data to generate segmented original PCM data;

S40: Send the segmented original PCM data to the cloud server, perform timbre learning through a preset timbre database, and match and identify the timbre of the audio data;

S50: Perform original text translation of the PCM data in the language required by the user in the cloud server, compare the translation result with the subtitle data, and use the subtitle data to synchronously correct the content and time of the translation result;

S60: Convert the corrected translation result into speech data of the same timbre according to the recognized timbre, and synthesize the speech data in synchronization with the video data and the subtitle data according to the time stamp, and synthesize a new program data stream for playing.

In one embodiment, the step S20 further includes:

After the audio data is acquired, the ambient sounds other than the human voice are filtered.

In one embodiment, the step S30 further includes: parsing punctuation marks in the subtitle data, acquiring a time position at each period, and segmenting the audio data according to the time position at the period.

In one embodiment, the step S60 further includes: comparing the amplitude of the converted voice data with the amplitude of the original audio data, so that the amplitude of the converted voice data is consistent with the amplitude of the original audio data.

A simultaneous interpretation output system for digital television programs provided by the embodiments of the present application includes:

a television program buffer module for controlling an audio and video terminal to buffer a television program data stream;

The data separation module separately parses and separates the video data, the audio data and the subtitle data from the buffered television program data stream, and marks the time stamp when separating, and marks the synchronization label for the three;

The audio segmentation module segments the audio data, and decodes the segmented audio data to generate segmented original PCM data;

The tone matching module sends the segmented original PCM data to the cloud server to learn the tone color through the preset tone color database, and matches the tone color of the recognized audio data;

The audio translation module converts the original PCM data into the language of the user's desired language in the cloud server, compares the translation result with the subtitle data, and uses the subtitle data to synchronously correct the content and time of the translation result;

The audio synthesis module converts the corrected translation result into voice data of the same timbre according to the recognized timbre, and synchronizes the voice data with the video data and the subtitle data according to the time stamp, and synthesizes a new program data stream for playing.

In one embodiment, the data separation module filters ambient sounds other than vocals after the audio data is acquired.

In one of the embodiments, the audio segmentation module parses the punctuation marks in the subtitle data, acquires the temporal position at each period, and segments the audio data according to the temporal position at the period.

In one embodiment, the audio synthesis module compares the amplitude of the converted voice data with the amplitude of the original audio data, so that the amplitude of the converted voice data is consistent with the amplitude of the original audio data.

An intelligent terminal provided by the embodiment of the present application includes:

At least one processor; and,

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform simultaneous interpretation of digital television programs as described above method.

The embodiment of the present application further provides a non-transitory computer readable storage medium storing computer executable instructions for causing a smart terminal to execute as described above The simultaneous interpretation method of digital TV programs.

The embodiment of the present application further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instruction is by a smart terminal When executed, the smart terminal is caused to perform the simultaneous interpretation output method of the digital television program as described above.

The method, system and intelligent terminal for simultaneous interpretation of digital television programs provided by the embodiments of the present application separate the buffered television program data streams into video, audio and subtitles, and then segment, timbre and translate the audio data. Processing and other processing, and using the subtitle data and time stamp for correction and synchronization processing, complete the simultaneous interpretation of the original audio data into the audio data of the user's desired language, and then broadcast The user is allowed to understand the audio of the TV program without watching the subtitles, which brings great convenience to the user to watch the TV program, and the user does not miss the screen content of the TV program, thereby greatly improving the user's viewing experience. .

DRAWINGS

1 is a flow chart of a method for simultaneous interpretation output of a digital television program in an embodiment;

2 is a structural diagram of a simultaneous interpretation output system of a digital television program in an embodiment;

FIG. 3 is a schematic diagram of a hardware structure of an intelligent terminal in an embodiment.

detailed description

In order to make the objects, technical solutions, and advantages of the present application more comprehensible, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting.

1 is a flow chart of a method for simultaneous interpretation output of a digital television program in an embodiment. As shown in FIG. 1, the method includes the following steps:

S10: Control the audio and video terminal to buffer the stored TV program data stream.

Since many TV programs are real-time programs, many of the TV program data streams are real-time streams. Therefore, in order to enable simultaneous interpretation of the TV programs, in this embodiment, it is first necessary to cache the TV program data streams, and buffer the time to the TV. The program data stream is processed.

S20: separately parsing and separating the video data, the audio data, and the caption data from the buffered TV program data stream, and marking the time stamp when separating, and marking the synchronization tag for the three.

After the cached television program data stream is stored, the video data, the audio data, and the subtitle data are separated for subsequent audio conversion. In this embodiment, in order to ensure that the reorganization can be synchronized after separation, the time stamps are marked when the three are separated, and the synchronization labels are marked for the three, thus ensuring subsequent synchronization operations.

Since the audio data includes a large number of environmental sounds in addition to the human voice, the ambient sound may interfere with the human voice. Further, the step further includes: after acquiring the audio data, the voice is removed. The external ambient sound is filtered.

S30: Segment the audio data, and decode the segmented audio data to generate segmented original PCM (an encoding format, also called pulse code modulation) data.

In order to ensure the integrity and rationality of the audio data statement, the audio data needs to be segmented, and the translation processing is also facilitated after segmentation. The audio data is decoded into original PCM data after segmentation so that it can be identified and processed.

Further, segmenting the audio data in the step is specifically: parsing the punctuation marks in the subtitle data, obtaining the time position at each period, and segmenting the audio data according to the time position at the period, so that according to the statement The integrity and consistency of the audio data is segmented.

S40: Send the segmented original PCM data to the cloud server for timbre learning through a preset timbre database, and match the timbre that identifies the audio data.

Due to the audio content, the tone of the audio is also an important parameter in the audio translation. The accurate translation of the tone can greatly ensure the effect of simultaneous interpretation. Therefore, in this embodiment, after the audio data is converted into PCM data, the audio is transmitted. To the front end for tone learning processing, using the preset tone database to match the tone in the PCM data, the most possible true restoration. The preset tone database is built by inputting sounds of different ages and genders.

S50: The original PCM data is translated in the language required by the user in the cloud server, and the translation result is compared with the subtitle data, and the subtitle data is used to synchronously correct the content and time of the translation result.

After the timbre learning is completed, since the original PCM data is uttered in a foreign language, it needs to be translated and translated into the language required by the user. Firstly, the original PCM data is translated into the text statement of the language required by the user in the cloud server. After the translation of the text statement, there may be a large error in the translation, so the translation result is compared with the subtitle data, and the subtitle data is used to The translation results are corrected for content and synchronized in time to eliminate errors in the content and time synchronization of the translation results.

After the translation result of the text translation is corrected and corrected, since the timbre data of the audio data has been obtained before, the speech result is combined with the recognized timbre, and the corrected translation result is converted into the speech data of the same timbre. The translated new audio data is finally synchronized with the video data and the subtitle data according to the time stamp, and the translated program data stream is played for playing, so that the simultaneous interpretation of the television program can be completed, so that the user can understand the television program. Audio to meet user needs.

In addition, in order to further improve the effect of simultaneous interpretation, the step further comprises: comparing the amplitude of the converted voice data with the amplitude of the original audio data, so that the amplitude of the converted voice data is consistent with the amplitude of the original audio data. .

The digital television program simultaneous voice output method separates the buffered television program data stream into video, audio and subtitle, and then performs segmentation, timbre recognition and translation processing on the audio data, and uses the subtitle data and time. The stamp is corrected and synchronized, and the original audio data is simultaneously translated into the audio data of the user's desired language, and then played to the user, so that the user can understand the audio of the television program without watching the subtitle, and the user can watch the television program. With great convenience, the user will not miss the screen content of the TV program, which greatly improves the user's viewing experience.

Meanwhile, the application also provides a simultaneous interpretation output system for digital television programs, as shown in FIG. 2, the system includes:

The TV program cache module 100 controls the audio and video terminal to buffer the stored TV program data stream.

Since many TV programs are real-time programs, the TV program data stream is a real-time stream. Therefore, in order to enable simultaneous interpretation of the TV program, in this embodiment, the TV program cache module 100 first needs to cache the TV program data stream. The television program data stream is processed by the cache time.

The data separation module 200 separately parses and separates the video data, the audio data, and the subtitle data from the buffered television program data stream, and marks the time stamp when separated, and marks the synchronization label for the three.

After the cached television program data stream is stored, the data separation module 200 separates the video data, the audio data, and the subtitle data for subsequent audio conversion. In this embodiment, in order to ensure that the reorganization can be synchronized after the separation, the data separation module 200 marks the time stamp when the three are separated, and marks the synchronization label for the three, thus ensuring subsequent synchronization operations.

In addition to the human voice, the audio data also includes a large number of environmental sounds, and the environmental sound will be human. The sound causes interference, so further, the data separation module 200 filters the ambient sounds other than the human voice after acquiring the audio data.

The audio segmentation module 300 segments the audio data and decodes the segmented audio data to generate segmented original PCM (an encoding format, also referred to as pulse code modulation) data.

In order to ensure the integrity and rationality of the audio data statement, the audio segmentation module 300 needs to segment the audio data, and facilitates the translation process after segmentation. The audio data is decoded into original PCM data after segmentation so that it can be identified and processed.

Further, the audio segmentation module 300 parses the punctuation marks in the subtitle data, obtains the time position at each period, and segments the audio data according to the time position at the period, so that the integrity and consistency of the statement are good. The audio data is segmented.

The timbre matching module 400 sends the segmented original PCM data to the cloud server for timbre learning through a preset timbre database, and matches the timbre of the recognized audio data.

Due to the audio content, the tone of the audio is also an important parameter in the audio translation. The accurate translation of the tone can greatly ensure the effect of simultaneous interpretation. Therefore, in this embodiment, after the audio data is converted into PCM data, the audio is transmitted. To the front end for timbre learning processing, the timbre matching module 400 uses the timbre database set in advance to match the timbre in the PCM data, the most likely true restoration. The preset tone database is built by inputting sounds of different ages and genders.

The audio translation module 500 performs the text translation of the original PCM data in the language required by the user in the cloud server, compares the translation result with the subtitle data, and uses the subtitle data to synchronously correct the content and time of the translation result.

After the timbre learning is completed, since the original PCM data is uttered in a foreign language, it needs to be translated and translated into the language required by the user. The audio translation module 500 first translates the original PCM data into a text statement of the language required by the user in the cloud server. After the translation of the text sentence is completed, the translation result may be compared with the subtitle data, and the translation result is compared. The subtitle data is used to correct the content of the translation, and the time synchronization is performed to eliminate the error of the translation result in content and time synchronization.

The audio synthesis module 600 converts the corrected translation result into the same sound according to the recognized timbre Color voice data, and the voice data is synchronized with the video data and the caption data according to the time stamp, and a new program data stream is synthesized for playing.

After the translation result of the text translation is corrected and corrected, since the timbre data of the audio data has been obtained before, the audio synthesis module 600 combines the recognized timbre to synthesize the translation result, and convert the corrected translation result into the same timbre. The voice data is obtained, and the translated new audio data is obtained. Finally, the time stamp is synthesized synchronously with the video data and the subtitle data, and the translated program data stream is played for playing, so that the simultaneous interpretation of the television program can be completed, so that the user can listen. Understand the audio of TV programs to meet user needs.

In addition, in order to further improve the effect of the simultaneous interpretation, the audio synthesis module 600 compares the amplitude of the converted speech data with the amplitude of the original audio data, so that the amplitude of the converted speech data is consistent with the amplitude of the original audio data.

The digital television program simultaneous voice output system separates the buffered television program data stream into video, audio and subtitles, and then performs segmentation, tone recognition and translation processing on the audio data, and uses the caption data and time. The stamp is corrected and synchronized, and the original audio data is simultaneously translated into the audio data of the user's desired language, and then played to the user, so that the user can understand the audio of the television program without watching the subtitle, and the user can watch the television program. With great convenience, the user will not miss the screen content of the TV program, which greatly improves the user's viewing experience.

The method and system for simultaneous interpretation of digital television program of the present application separates the buffered television program data stream into three parts: video, audio and subtitle, and then processes the audio data by segmentation, timbre recognition and translation processing, and uses subtitles. The data and time stamp are corrected and synchronized, and the original audio data is simultaneously translated into the audio data of the user's desired language, and then played to the user, so that the user can understand the audio of the television program without watching the subtitle, and view the audio for the user. The TV program brings great convenience, and the user does not miss the screen content of the TV program, thereby greatly improving the user's viewing experience.

FIG. 3 is a schematic diagram of a hardware structure of an intelligent terminal according to an embodiment of the present application. The smart terminal 700 can perform the simultaneous interpretation output method of the digital television program described in any of the foregoing method embodiments. The smart terminal 700 can include, but is not limited to, a set top box, a television, a mobile phone, a tablet, and the like.

Specifically, referring to FIG. 3, the smart terminal 700 includes:

One or more processors 701 and a memory 702 are exemplified by a processor 701 in FIG.

The processor 701 and the memory 702 may be connected by a bus or other means, as exemplified by a bus connection in FIG.

The memory 702 is used as a non-transitory computer readable storage medium, and can be used for storing non-transitory software programs, non-transitory computer executable programs, and modules, and corresponding to the digital television program simultaneous translation output method in the embodiment of the present application. Program instructions/modules (for example, the television program cache module 100, the data separation module 200, the audio segmentation module 300, the tone color matching module 400, the audio translation module 500, and the audio synthesis module 600 shown in FIG. 2). The processor 701 performs various functional applications and data processing of the digital television program simultaneous translation output system by running non-transitory software programs, instructions, and modules stored in the memory 702, ie, the numbers implementing any of the above method embodiments Simultaneous translation output method for TV programs.

The memory 702 can include a storage program area and a storage data area, wherein the storage program area can store an operating system, an application required for at least one function; the storage data area can be stored according to the use of the digital television program simultaneous translation output system. Data, etc. Moreover, memory 702 can include high speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 702 can optionally include memory remotely located relative to processor 701 that can be connected to smart terminal 700 over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The one or more modules are stored in the memory 702, and when executed by the one or more processors 701, perform a digital television program simultaneous translation output method in any of the above method embodiments, for example, performing the above The described method steps S10 to S60 in Fig. 1 implement the functions of the modules 100-600 in Fig. 2.

The embodiment of the present application further provides a non-transitory computer readable storage medium storing computer executable instructions executed by one or more processors, for example, Executed by a processor 701 in FIG. 3, the one or more processors may perform the digital television program simultaneous translation output method in any of the foregoing method embodiments, for example, The functions of the modules 100-600 of Figure 2 are implemented in the method steps S10 to S60 of Figure 1 described above.

The embodiment of the present application further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when When the program instruction is executed by the smart terminal, the smart terminal is configured to perform the digital TV program simultaneous translation output method in any of the foregoing method embodiments, for example, performing the method steps S10 to S60 in FIG. 1 described above to implement the method in FIG. The functionality of the module 100-600.

The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, ie may be located A place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

Through the description of the above embodiments, those skilled in the art can clearly understand that the various embodiments can be implemented by means of software plus a general hardware platform, and of course, by hardware. A person skilled in the art can understand that all or part of the process of implementing the above embodiments can be completed by a computer program to instruct related hardware, and the program can be stored in a non-transitory computer readable storage medium. The program, when executed, may include the flow of an embodiment of the methods as described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

The above products can perform the methods provided by the embodiments of the present application, and have the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiments of the present application.

The above is only the preferred embodiment of the present application, and is not intended to limit the application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are included in the scope of the present application. Inside.

Claims

A simultaneous interpretation output method for digital television programs, characterized in that the method comprises the following steps:

S10: controlling the audio and video terminal to buffer the stored television program data stream;

S20: parsing and separating the video data, the audio data, and the subtitle data from the buffered television program data stream, and marking the time stamp when separating, marking the synchronization label for the three;

S30: segmenting the audio data, and decoding the segmented audio data to generate segmented original PCM data;

S40: Send the segmented original PCM data to the cloud server, perform timbre learning through a preset timbre database, and match and identify the timbre of the audio data;

S50: Perform original text translation of the PCM data in the language required by the user in the cloud server, compare the translation result with the subtitle data, and use the subtitle data to synchronously correct the content and time of the translation result;

S60: Convert the corrected translation result into speech data of the same timbre according to the recognized timbre, and synthesize the speech data in synchronization with the video data and the subtitle data according to the time stamp, and synthesize a new program data stream for playing.
The method for outputting a simultaneous interpretation of a digital television program according to claim 1, wherein the step S20 further comprises:

After the audio data is acquired, the ambient sounds other than the human voice are filtered.
The method for outputting a simultaneous interpretation of a digital television program according to claim 2, wherein said step S30 further comprises: parsing punctuation marks in the subtitle data, obtaining a time position at each period, according to a time position at the period Segment the audio data.
The method for outputting a simultaneous interpretation of a digital television program according to claim 3, wherein the step S60 further comprises: comparing the amplitude of the converted voice data with the amplitude of the original audio data to make the converted voice The amplitude of the data is consistent with the amplitude of the original audio data.
A simultaneous interpretation output system for digital television programs, characterized in that it comprises:

a television program buffer module for controlling an audio and video terminal to buffer a television program data stream;

The data separation module separately parses and separates the video data, the audio data and the subtitle data from the buffered television program data stream, and marks the time stamp when separating, and marks the synchronization label for the three;

The audio segmentation module segments the audio data, and decodes the segmented audio data to generate segmented original PCM data;

The tone matching module sends the segmented original PCM data to the cloud server to learn the tone color through the preset tone color database, and matches the tone color of the recognized audio data;

The audio translation module converts the original PCM data into the language of the user's desired language in the cloud server, compares the translation result with the subtitle data, and uses the subtitle data to synchronously correct the content and time of the translation result;

The audio synthesis module converts the corrected translation result into voice data of the same timbre according to the recognized timbre, and synchronizes the voice data with the video data and the subtitle data according to the time stamp, and synthesizes a new program data stream for playing.
The simultaneous interpretation output system for digital television programs according to claim 5, wherein the data separation module filters ambient sounds other than vocals after the audio data is acquired.
The simultaneous interpretation output system for digital television programs according to claim 6, wherein the audio segmentation module parses punctuation marks in the subtitle data to obtain a time position at each period, according to the time position at the period The audio data is segmented.
The simultaneous interpretation output system for digital television programs according to claim 7, wherein the audio synthesis module compares the amplitude of the converted voice data with the amplitude of the original audio data to make the converted voice data. The amplitude is consistent with the amplitude of the original audio data.
An intelligent terminal, comprising:

At least one processor; and,

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any of claims 1-4 The simultaneous interpretation method of digital TV programs.
A non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium stores computer executable instructions for causing an intelligent terminal to perform the claims 1-4 A simultaneous interpretation output method for digital television programs according to any one of the preceding claims.
A computer program product, comprising: a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a smart terminal, The smart terminal is configured to perform the simultaneous interpretation output method of the digital television program according to any one of claims 1-4.