CN115396690A - Audio and text combination method and device, electronic equipment and storage medium - Google Patents

Abstract

Embodiments of the present disclosure provide an audio and text combination method, device, electronic equipment, and storage medium. The method for combining audio and text includes: performing text recognition on key frames in a video file to obtain a first subtitle set; performing subtitle switching detection on the first subtitle set to obtain target subtitles and the time when subtitle switching occurs in the target subtitles stamp; based on the time stamp, intercept the audio from the audio file of the video file to obtain the target audio; based on the target subtitle and the target audio, construct a combination pair of text and audio. In the embodiment of the present disclosure, the matching degree between audio and text can be improved.

Description

Audio and text combination method, device, electronic device and storage medium

technical field

The present disclosure relates to the technical field of artificial intelligence, and in particular to an audio and text combination method, device, electronic equipment and storage medium.

Background technique

Speech recognition technology is a kind of artificial intelligence technology, which is widely used in many fields such as voice input, voice interaction, voice search, voice control, and voice subtitles. Especially under the impact of the current epidemic, contactless voice interaction and control highlights the value of voice recognition technology. Since speech recognition technology is based on deep learning algorithms, it relies on massive and high-quality training data to improve the accuracy of the algorithm, making the quality of speech recognition ideal.

The training data generation method of existing speech recognition algorithms is generally as follows: obtain video resources with subtitles, such as audiobooks, on-site commentary, documentaries, dramas, interviews, news, variety shows, self-media and many other video resources; OCR (optical character recognition) text recognition technology extracts subtitles and subtitle timestamps from the video; matches the audio segment corresponding to the subtitle from the video through the timestamp, and forms the subtitle and the audio segment into a pair of training samples for <text, audio> .

However, the existing subtitle extraction schemes all use OCR text recognition technology to extract subtitles in videos. This method only relies on unilateral information of video content, so that the accuracy of subtitle extraction depends on the accuracy of OCR recognition technology. When there is a complex background in the video or the position of the subtitle is not fixed, the OCR recognition effect is not good, so when matching the <text, audio> pair, it is easy to misplace the subtitle and audio time information, and the content of the subtitle and the audio content are not aligned. Happening. Furthermore, it affects the recognition accuracy of the speech recognition algorithm.

Contents of the invention

Embodiments of the present disclosure provide an audio and text combination method, device, electronic device, and storage medium, so as to solve or alleviate one or more technical problems in the prior art.

As a first aspect of the embodiments of the present disclosure, the embodiments of the present disclosure provide a method for combining audio and text, including:

Carry out text recognition to the key frame in the video file, obtain the first subtitle set;

Performing subtitle switching detection on the first subtitle set to obtain the target subtitle and the timestamp when subtitle switching occurs in the target subtitle;

Based on the timestamp, intercept the audio from the audio file of the video file to obtain the target audio;

Based on the target subtitle and the target audio, a combined pair of text and audio is constructed.

As a second aspect of the embodiments of the present disclosure, the embodiments of the present disclosure provide an audio and text combination device, including:

A text recognition module is used to carry out text recognition to key frames in the video file to obtain the first subtitle set;

A subtitle switch detection module, configured to perform subtitle switch detection on the first subtitle set, to obtain the target subtitle and the timestamp when the target subtitle is switched;

The target audio interception module is used to intercept the audio from the audio file of the video file based on the timestamp to obtain the target audio;

An audio-text combination module, configured to construct a combination pair of text and audio based on the target subtitle and the target audio.

As a third aspect of the embodiments of the present disclosure, the embodiments of the present disclosure provide an electronic device, including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can execute the audio and text combination method provided by the embodiments of the present disclosure .

As the fourth aspect of the embodiments of the present disclosure, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to make the computer execute the combination of audio and text provided by the embodiments of the present disclosure method.

As a fourth aspect of the embodiments of the present disclosure, the embodiments of the present disclosure provide a computer program product, including a computer program. When the computer program is executed by a processor, the method for combining audio and text provided by the embodiments of the present disclosure is implemented.

In the technical solution provided by the embodiments of the present disclosure, text recognition is performed on key frames in a video file to obtain a first subtitle set, and subtitle switching detection is performed on the first subtitle set to obtain target subtitles and subtitle switching occurs in the target subtitles the timestamp of . Thereby, based on the time stamp, the audio is intercepted from the audio file of the video file, and the accurately determined target audio can be obtained, so that the matching of the combined pair of text and audio constructed based on the target subtitle and the target audio High degree, to avoid misalignment of subtitle and audio time information, misalignment of subtitle content and audio content.

The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present disclosure will be readily apparent by referring to the drawings and the following detailed description.

Description of drawings

In the drawings, unless otherwise specified, the same reference numerals designate the same or similar parts or elements throughout the several drawings. The drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to the present disclosure and should not be taken as limiting the scope of the present disclosure.

Fig. 1 is a flowchart of an audio and text combination method according to an embodiment of the present disclosure;

FIG. 2 is a flowchart of a subtitle processing method according to an embodiment of the present disclosure;

FIG. 3A is a schematic diagram of a key frame according to an embodiment of the present disclosure;

FIG. 3B is a schematic diagram of a key frame according to another embodiment of the present disclosure;

Fig. 4 is a schematic diagram of a key frame according to another embodiment of the present disclosure;

Fig. 5 is a flow chart of the construction process of the training sample of the speech recognition model of an embodiment of the present disclosure;

FIG. 6 is a flowchart of a subtitle file generation process according to an embodiment of the present disclosure;

FIG. 7 is a flow chart of subtitle verification, switching detection and error correction according to an embodiment of the present disclosure;

Fig. 8 is a schematic diagram of an anatomy of a pictographic character according to an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a subtitle file according to an embodiment of the present disclosure;

Fig. 10 is a schematic diagram of an audio and text combination device according to an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of an electronic device according to an embodiment of the present disclosure;

Detailed ways

In the following, only some exemplary embodiments are briefly described. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present disclosure. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature and not restrictive.

Fig. 1 is a flowchart of a method for combining audio and text provided by an embodiment of the present disclosure. As shown in Figure 1, the audio and text combination method may include the following steps:

S110, performing text recognition on key frames in the video file to obtain a first subtitle set;

S120, performing subtitle switching detection on the first subtitle set, and obtaining the target subtitle and the time stamp when subtitle switching occurs in the target subtitle;

S130, based on the time stamp when subtitle switching occurs in the target subtitle, intercept the audio from the audio file of the video file to obtain the target audio;

S140. Construct a combined pair of text and audio based on the target subtitle and the target audio.

In this example, text recognition is performed on key frames in the video file to obtain a first subtitle set, and subtitle switch detection is performed on the first subtitle set to obtain a target subtitle and a time stamp when the target subtitle is switched. Thereby, based on the time stamp, the audio is intercepted from the audio file of the video file, and the accurately determined target audio can be obtained, so that the matching of the combined pair of text and audio constructed based on the target subtitle and the target audio High degree, to avoid misalignment of subtitle and audio time information, misalignment of subtitle content and audio content.

Exemplarily, the video file includes multiple image frames played in time sequence and corresponding audio. Video files can include audiobooks, on-site commentary, documentaries, dramas, interviews, news, variety shows, self-media and other video resources. Video files can be short videos or longer videos.

Exemplarily, in the above step S110, text recognition may be performed on all key frames in the video file, or text recognition may be performed on some key frames. A keyframe can be any image frame in a video file, or an image frame with text.

Exemplarily, the text recognition may adopt OCR recognition technology.

Exemplarily, the first subtitle set includes multiple subtitles, and each subtitle may come from a different key frame. Subtitles may be subtitles with the same content or subtitles with different content.

Exemplarily, for a subtitle, it may appear in multiple consecutive key frames. When it is different from the subtitle of the previous frame, or different from the subtitle of the next frame, it can be considered that subtitle switching occurs . Therefore, subtitle switch detection may be to use the subtitle similarity of two adjacent frames to detect whether a subtitle switch occurs. If the subtitle similarity between two adjacent frames is greater than the set subtitle similarity threshold, it is determined that subtitle switching has not occurred. If the subtitle similarity between two adjacent frames is smaller than the set subtitle similarity threshold, it is determined that subtitle switching occurs. If subtitle switching occurs, the time stamp at which subtitle switching occurs may be determined based on the time stamps of two adjacent frames. For example, the time stamp of the first frame of the two adjacent frames is used as the time stamp of subtitle switching, or the time stamp of the second frame of the two adjacent frames is used as the time stamp of subtitle switching.

Exemplarily, the time stamp at which subtitle switching occurs includes a start time stamp and an end time stamp. The subtitles in each key frame from the start time stamp to the end time stamp have the same subtitle content, that is, the subtitle similarity of two adjacent frames is greater than the set subtitle similarity threshold.

Exemplarily, the constructed combination pair of text and audio can be applied to the training of the speech recognition model.

In some embodiments, in the above step S110, the process of performing text recognition on key frames in the video file to obtain the first subtitle set includes operations such as checking whether the text is a subtitle, filtering non-subtitle text, and the like.

Exemplarily, FIG. 2 is a flowchart of a subtitle processing method provided by an embodiment of the present disclosure. As shown in Figure 2, the subtitle processing method may include the following steps:

S210, performing text recognition on key frames in the video file to obtain candidate subtitle texts;

S220, in the video file, determine the verification audio corresponding to the candidate subtitle text;

S230, when the relationship between the candidate subtitle text and the verification audio meets the set condition, determine that the candidate subtitle text is the subtitle text of the video file;

S240. Obtain the subtitle text of the video file to obtain a first subtitle set.

In this embodiment, the verification text of speech recognition is used to verify the candidate subtitle text obtained by text recognition, and determine whether the candidate subtitle text is a real subtitle, thereby improving the accuracy of subtitle recognition, so that in the subsequent construction When the combination is composed of text and audio, it can avoid the misalignment of text and audio time information, and the misalignment of text content and audio content.

Exemplarily, in the above step S210, text recognition may be performed on all key frames in the video file, or text recognition may be performed on some key frames. A keyframe can be any image frame in a video file, or an image frame with text.

Exemplarily, performing text recognition on the key frame in the above step S210 may be an OCR text recognition technology. When performing text recognition on the key frame, one or more texts in the key frame can be obtained, such as a text block or a text line. In this case, some texts may not be subtitles, and these texts can be used as candidate subtitles. Furthermore, an audio verification is performed on the candidate subtitles to accurately determine whether the candidate subtitles are real subtitles.

Exemplarily, text recognition is performed on key frames in the video file to obtain at least one text, and according to the subtitle area of the video file and the position information of each text in the key frame, candidate subtitles are determined among multiple texts in the key frame text to further determine whether the candidate subtitle text is a subtitle. For example, if the position information of the text matches the subtitle area, then these texts are considered to be subtitles, and if the position information of the text does not match the subtitle area, then these texts are not necessarily considered to be subtitles, and it is necessary to further check whether they are subtitles. The position information may include the position area of the text in the key frame, for example, the text is framed by a rectangle, and the coordinates of the four corners of the rectangle are the position area of the text in the key frame.

Exemplarily, the candidate subtitle text is text that has not been determined whether it is a subtitle, for example, it may be a subtitle, or it may be background text or watermark text in a key frame.

Exemplarily, as shown in FIG. 3A and FIG. 3B , this is a scene where the teacher Actor1 is teaching. The text in the text area pointed by the teacher Actor1 in the figure is the background text, and the two text blocks below the text area are "This is the poem of Li Bai, a poet of the Tang Dynasty" and "The poet uses the deep lake as a metaphor for Wang Lun and his Friendship uses exaggeration" is the subtitle. If OCR recognition is used, the background text and the subtitles are recognized at the same time. At this time, it is impossible to know which ones are subtitle texts and which ones are background texts. Therefore, by using the method provided by the embodiment of the present disclosure, subtitles can be accurately identified and background text can be filtered out.

Exemplarily, in the above step 220, based on the time stamp corresponding to the candidate subtitle text, the corresponding audio segment can be intercepted from the audio in the video file to obtain the verification audio. The time stamp may be the time stamp when the candidate subtitle text is switched in the video file, that is, the time when the candidate subtitle text appears for the first time and the last time it appears in the sequential image frames. Therefore, the target audio corresponding to the candidate subtitle text can be accurately intercepted, and the accuracy of subsequent subtitle verification can be improved.

In some embodiments, whether the candidate subtitle text is a subtitle can be determined by using the degree of coincidence or confidence between the candidate subtitle text and its corresponding verification text.

Exemplarily, the above step S230 may include: performing speech recognition on the verification audio to obtain the verification text; when the degree of overlap between the candidate subtitle text and the verification text is greater than the set coincidence degree threshold, determine the candidate subtitle Text is the subtitle text of the video file.

Exemplarily, the foregoing speech recognition may adopt an ASR (Automatic Speech Recognition, automatic speech recognition) technology.

Exemplarily, the coincidence degree refers to the ratio between the same character and all characters between two character strings, or the ratio between the same character and one of the character strings. The ratio can be a ratio between the number of characters, or a ratio between the lengths of characters.

Exemplarily, the coincidence degree m between the candidate subtitle text and its corresponding verification text is m=len(string(ocr)&string(asr))/len(string(ocr), where string(ocr) is the candidate subtitle text, tring(asr) is the verification text corresponding to the candidate subtitle text, len(string(ocr)&string(asr)) is the length of the character intersection between the candidate subtitle text and the verification text, len(string(ocr) is The length of the candidate subtitle text. The character intersection includes the set of identical characters between the candidate subtitle text and the verification text.

In this example, it may be determined whether the candidate subtitle text is a subtitle by using the coincidence degree between the candidate subtitle text and the verification text identified by the corresponding audio. Therefore, the accuracy of subtitle recognition is improved.

In some embodiments, when a large number of subtitles are identified on a video file, some obvious non-subtitle texts such as watermarks may be pre-filtered.

Exemplarily, in the above step S210, text recognition is performed on key frames in the video file to obtain candidate subtitle texts, including:

performing text recognition on key frames in the video file to obtain a first text set;

determining non-subtitle text based on the first set of text;

Filter the non-subtitle text in the first text set to obtain the second text set;

Based on the second set of text, candidate subtitle text is determined.

In this example, pre-filtering non-subtitle text is beneficial to improve the efficiency of subsequent subtitle recognition.

Exemplarily, before text recognition is performed on each key frame in the video file, key frames are extracted from the video file. Text detection is performed for each image frame in the video file, and the frame difference method is used to compare whether the two frames before and after are repeated. If it is repeated, the redundant frame is deleted, and finally multiple key frames are obtained, and each key frame contains text. Whether the two frames of images are repeated frames can be determined by comparing the similarity between the texts in the two frames of images before and after, such as the edit distance. The model for text detection on image frames can be implemented using the RCNN deep learning model. In addition, in order to increase the speed of extracting key frames, key frames can be acquired at set intervals, for example, 3 frames per second are extracted as key frames, and one frame is extracted as key frames every 8 seconds.

Exemplarily, by performing text recognition on a key frame, the text information and position information of each text in the key frame can be obtained. Each text can be intercepted in the form of text block or text line. For example, a text recognition result for a keyframe can be labeled as:

[(text1,(xmin,xmax,ymin,ymax)),(text2,(xmin,xmax,ymin,ymax)),...];

Among them, text1 is a text, text2 is another text; xmin, xmax represent the minimum abscissa and maximum abscissa of the text block in the current key frame; ymin, ymax represent the minimum ordinate of the text block in the current key frame and the maximum ordinate. As shown in FIG. 4 , it shows the position information of the text block "study hard, improve every day" in the current frame.

In some embodiments, the non-subtitle text includes background text such as watermarks, logos, or advertisements. For example, in some TV dramas and variety show nodes, there will be background information such as station logos or advertisements. As shown in Fig. 4, "XXTV drama channel" in Fig. 4 is a station logo. These background information often appear in a fixed position in the video, and exist in the entire playing time of the video, that is, in all key frames of the video. Therefore, based on the position information of each text in the first text set, the frequency of occurrence of texts with the same position can be counted, and the text with the frequency of occurrence higher than the set threshold can be determined as non-subtitle text.

Exemplarily, the above-mentioned determination of the non-subtitle text based on the position information of each text in the first text set may include: determining the occurrence frequency of text with the same position based on the position information of each text in the first text block set; When the frequency of occurrence is greater than the set frequency threshold, the text with the same position is determined as non-subtitle text.

It should be noted that there may be slight differences in the position coordinate information of the same text between frames before and after or between different frames. For example, the position coordinate information identified in the two frames before and after "" study hard and improve every day" in Fig. 4 can be (125,512,678,780) and (126,513,679,781). Therefore, it can The coordinate information of these two positions is normalized and regarded as the same position.

Exemplarily, when it is determined based on the position information of two texts belonging to different key frames that the position distance between the two texts is less than a predetermined distance threshold, it is determined that the two texts are texts in the same position.

Exemplarily, the position of text block 1 in key frame a is (x1, y1); the position of text block 2 in key frame b is (x2, y2). If ||x1-x2||<a,||y1-y2||<b, then text block 1 and text block 2 are texts with the same position.

Exemplarily, the frequency of occurrence of each text with the same position is counted to obtain the frequency of occurrence of each text with the same position. The text with the highest frequency of occurrence is determined as the non-subtitle text, or the text with the frequency of occurrence greater than 80% or 70% of the total number of key frames in the video file is determined as the non-subtitle text.

In some embodiments, the second text set is obtained after filtering the non-subtitle text in the first text set. At this time, all the texts in the second text set may be used as candidate subtitle texts for audio verification to obtain the subtitle text of the video file. Alternatively, the texts in the second text set are screened, and the screened texts are used as candidate subtitle texts.

In some embodiments, candidate subtitle texts may be screened out from the second text set by using the subtitle area.

In some embodiments, the method of determining the subtitle area can be: after filtering the non-subtitle text, the remaining text can be sorted by the frequency of occurrence of the position, and the frequency of occurrence is greater than the set threshold, for example, the highest frequency text, the location area is determined as the subtitle area of the video text.

In some embodiments, the subtitle area may be used to determine whether the text in the second text set after filtering the non-subtitle text is a subtitle. For example, if the position of the text matches the subtitle area, the text can be determined as a subtitle. If the position of the text does not match the subtitle area, it cannot be determined whether the text is a subtitle, and further verification is required.

Exemplarily, the above-mentioned determination of candidate subtitle text based on the second text set includes:

determining a first subtitle area based on the position information of each text in the second text set;

For each text in the second text set, if the matching degree between the location information of the text and the first subtitle area is less than a set matching threshold, the text is determined to be a candidate subtitle text.

In this example, the candidate subtitle text is text for which it is not possible to determine whether it is a subtitle, and it needs to be further verified by using the verification text obtained through speech recognition, so as to improve the accuracy of subtitle recognition.

Exemplarily, the above-mentioned determination of the first subtitle area based on the position information of each text in the second text set may include: sorting the frequency of occurrence of the text with the same position in the second text set, and sorting the text with the highest frequency of occurrence The location area is determined as the first subtitle area.

Exemplarily, as shown in FIG. 3A and FIG. 3B , they are two key frames in a video of a teacher giving a lecture. Background subtitles exist in key frames and are multi-line subtitles. The text in the text area that the teacher Actor1 is pointing at is the background text. Using the subtitle area determined above to verify the text area pointed by the teacher Actor1, it can be found that the subtitle area does not match the text area pointed by the teacher Actor1. Suppose the subtitle area is recorded as (Xmin, Xmax, Ymin, Ymax), and the text area pointed by Actor1 is (Xminc, Xmaxc, Yminc, Ymaxc), if Yminc<Ymin or Ymaxc>Ymax, the text corresponding to the text area is a subtitle Text outside the region cannot be identified as subtitles. However, subtitles may be lost if the text area is directly considered not to be subtitles. Therefore, this kind of text is determined as a candidate subtitle text, and the candidate subtitle text is verified by using the verification text obtained by speech recognition corresponding to the audio, so as to determine whether the text is a subtitle. Therefore, on the one hand, the loss of subtitles can be avoided, and on the other hand, the accuracy of subtitle recognition can be improved.

Since subtitles can be identified by using the subtitle area, the first subtitle set can be quickly obtained by using the subtitle area to identify the first text set or the second text set.

Exemplarily, the above-mentioned determination of candidate subtitle texts based on the second text set further includes:

For each text in the second text set, if the matching degree between the location information of the text and the first subtitle area is greater than a set matching threshold, it is determined that the text is the subtitle text of the video file.

Exemplarily, in the above step S210, text recognition is performed on the key frames of the video file to obtain the first subtitle set, which may include:

performing text recognition on each key frame in the video file to obtain a third text set;

determining a second subtitle area based on the position information of each text in the third text set;

For each text in the third text set, if the matching degree between the location information of the text and the second subtitle area is greater than a set matching threshold, it is determined that the text is the subtitle text of the video file.

In this example, by using the subtitle area to determine the subtitle, the subtitle can be determined quickly and accurately, and the efficiency of subtitle recognition can be improved.

Exemplarily, text recognition is performed on each key frame in the video file to obtain a third text set, and the above-mentioned method for filtering non-subtitle text can be used to filter the non-subtitle text in the third text set to obtain the filtered third text set. For details, reference may be made to the filtering process of the non-subtitle text and the recognition process of the non-subtitle text in the aforementioned second text set, which will not be described in detail here.

Exemplarily, the process of determining the second subtitle area may be the same as or similar to the aforementioned process of determining the first subtitle area. For details, reference may be made to the aforementioned content, which will not be described in detail here.

Exemplarily, the first subtitle area and the second subtitle area may be the same area, or may be different areas.

In some embodiments, after the first subtitle set is obtained, operations such as subtitle switching detection and deduplication may be performed on the first subtitle set, so as to avoid duplicate text and audio combination pairs.

Exemplarily, in the above step S120, it may include:

Based on the similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, determine the timestamp of subtitle switching, and use the timestamp to divide the first subtitle set to obtain a plurality of second subtitle sets;

Deduplicating the second subtitle set to obtain the target subtitle;

Based on the time stamp of the second subtitle set, determine the time stamp at which subtitle switching occurs for the target subtitle.

In this example, subtitle switch detection is performed on the first subtitle set to obtain multiple second subtitle sets, and each subtitle set includes multiple subtitles with the same content. Therefore, each second subtitle set can be deduplicated to obtain Target subtitles for combined text and audio pairs, avoiding duplicate text and audio combined pairs.

Exemplarily, before subtitle switch detection is performed on the first subtitle set, subtitles belonging to the same key frame are combined. For example, a key frame may have two lines of subtitles, and these two lines of subtitles can be combined into one subtitle. In this way, subsequent subtitle switching detection can be facilitated.

Exemplarily, in the first subtitle set, the similarity between any two subtitle texts belonging to adjacent key frames is calculated, and when the similarity is greater than the set first similarity threshold, the subtitles of the adjacent key frames are The text is determined as subtitles with the same content; when the similarity is less than the set similarity threshold, the two subtitle texts of the adjacent key frames are determined as subtitles with different content, that is, subtitle switching occurs, and the two The time stamp corresponding to one of the key frames is determined as the time stamp of subtitle switching.

After obtaining all the time stamps in which subtitle switching occurs in the first subtitle set, use the time stamp as a dividing line, divide the subtitles corresponding to the key frames whose time stamps belong to the two dividing lines into the same subtitle set, and obtain multiple second subtitles set.

Exemplarily, the second subtitle set includes multiple subtitles. The plurality of subtitles are subtitles having the same content.

Exemplarily, the start time stamp and end time stamp corresponding to the second subtitle set are time stamps when subtitle switching occurs for the target subtitle.

In some embodiments, although the similarities between subtitle texts belonging to adjacent key frames in the second subtitle set are greater than the first similarity threshold, they are not necessarily identical subtitles, which indicates that there may be text in the two subtitles. In the case of a recognition error, that is, there is a word that is a wrong word. Therefore, before performing deduplication on the second subtitle set, the typo-existing subtitles in the second subtitle set can be corrected.

In order to improve the efficiency of subtitle error correction, the similarity between any two subtitles in the second subtitle set can be used to determine the subtitle with typo, and correct the subtitle with typo.

Exemplarily, before deduplication is performed on the second subtitle set, the above method further includes:

When the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than the first similarity threshold and smaller than the second similarity threshold, text error correction is performed on the first subtitle text and the second subtitle text.

In this example, the time stamp of subtitle switching is used to divide the subtitle sets that belong to the same subtitle content, and based on the similarity between the subtitles in each subtitle set, the subtitles that need text error correction are determined to improve subtitle error correction. s efficiency.

Exemplarily, the above similarity may use edit distance to calculate the similarity of two texts. Edit distance, also called Levenshtein distance, is a quantitative measurement of the degree of difference between two character strings (such as English characters). The measure is the minimum number of operations required to turn one string into another.

Exemplarily, the first similarity threshold may be 90%, and the second similarity threshold may be 100%.

Exemplarily, the first similarity threshold may be 95%, and the second similarity threshold may be 99.9%.

In some embodiments, the subtitle text for text error correction can be masked with wrong words, and based on the masked subtitle text and masked words, the target predicted word can be determined, and then the masked word in the subtitle text can be corrected by using the target predicted word .

Exemplarily, the text error correction of the first subtitle text and the second subtitle text includes:

In the first subtitle text and the second subtitle text, the words with the same position and different words are determined as shielding words;

In the first subtitle text and the second subtitle text, the masked words are shielded to obtain the third subtitle text;

Based on the third subtitle text and masked words, determine the target prediction word;

Based on the target predicted words, the masked words in the first subtitle text and the second subtitle text are corrected.

In this example, the subtitle text and the masked words are obtained after masking the wrong words, and the target prediction word is determined, so that the correct words can be obtained accurately.

Exemplarily, the above-mentioned first subtitle text and second subtitle text may be subtitle texts of two adjacent frames.

Exemplarily, assuming that the first subtitle text is "There is a lot of traffic at the current intersection, you should be careful", and the second subtitle text is "The traffic at the current intersection is heavy, you should be careful", then "people" and "Enter" is two words with the same position but different phonetics, which are shielding words.

Exemplarily, the target prediction word may include one or more.

Exemplarily, semantic analysis may be performed on the third subtitle text to obtain the predicted word set and the confidence degree of each predicted word in each predicted word set, and then use the confidence degree of each predicted word, each predicted word and the masked word The relationship between to determine the target prediction word.

Exemplarily, the pre-trained language model BERT (BidirectionalEncoder Representation from Transformers) in natural language processing can be used to perform semantic analysis on the third subtitle text to obtain the predicted word set and the confidence of each predicted word in each predicted word set.

In practical applications, the BERT model is called a MASK (covered) language model, that is, its main task in the training phase is to predict the elements that are masked. For example, before entering the text into the BERT model, mask the suspected wrong words. For example, MASK is performed on the above-mentioned "person" and "entry", and the third subtitle text is obtained as "the current [MASK] traffic is very heavy, so be careful". Then, use the BERT model to predict what the masked word is. The set of candidate words in the MASK position predicted by BERT, and the confidence of each word.

In some embodiments, the target predicted word may be determined in combination with the confidence of each predicted word and the edit distance between each predicted word and the masked word.

Exemplarily, the determination of the target prediction word based on the third subtitle text and the masked word includes:

Carrying out semantic analysis to the third subtitle text to obtain the predicted word set and the confidence degree of each predicted word in the predicted word set;

Determine the quality score of each predicted word based on the confidence of each predicted word and the edit distance between each predicted word and the masked word;

Based on the quality score of each predicted word, the target predicted word is determined in the predicted word set.

In this example, combining the confidence of each predicted word and the edit distance between each predicted word and the masked word, determining the target predicted word among multiple predicted words can further improve the prediction accuracy.

Exemplarily, the predicted word and the confidence of the predicted word can be predicted by a model. Confidence refers to the degree of credibility that the predicted word is the real text in the masked position in the third subtitle text.

Exemplarily, the above-mentioned semantic analysis of the third subtitle text may use a BERT model.

For example, assume that the words in the MASK position predicted by the BERT module are: "car" and "person", where the confidence p of "car" is 0.8123, and the edit distance s between the masked word "in" is 0.012, The confidence p of "person" is 0.2313, and the edit distance s from the IDS of "in" is 0.9342. Then, summing the disposition degree and edit distance, the quality score of "car" is 0.8243, and the quality score of "person" is 1.1655. Therefore, the predicted word "person" can be selected as the target predicted word.

In some embodiments, it is possible to decide whether to modify based on the quality score of the target predicted word and the edit distance between the target predicted word and the masked word, so as to avoid erroneous modification.

Exemplarily, based on the target predicted word, the masked words in the first subtitle text and the second subtitle text are corrected, including:

When the quality score of the target predicted word is greater than the set quality score threshold, and the edit distance between the target predictive word and the masked word is greater than the set edit distance threshold, the first subtitle text and the second subtitle text The masked words are changed to target predicted words.

Exemplarily, when the quality score of the target predicted word is less than the set quality score threshold or the edit distance between the target predictive word and the masked word is less than the set edit distance threshold, the first subtitle text and the second subtitle text Blocked words in the text are not modified.

In this example, whether to modify is decided based on the quality score of the target predicted word and the edit distance between the target predicted word and the masked word, which can avoid mistaken modification.

Exemplarily, when the quality score of the target predicted word is greater than 1 and the edit distance between the target predicted word and the masked word is greater than 0.5, the masked word is corrected; otherwise, no modification is made and the subtitle text remains unchanged.

In some embodiments, after performing text error correction on the second subtitle set, deduplication may be performed on the second subtitle set to obtain the target subtitle.

Exemplarily, the above-mentioned deduplication of the second subtitle set to obtain the target subtitle may include:

When the similarity between any two subtitle texts in the second subtitle set is greater than the third similarity threshold, deduplication is performed on the second subtitle set to obtain the target subtitle.

In this example, if the similarity between any two subtitle texts in the second subtitle set is greater than the third similarity threshold, it means that the subtitles in the second subtitle set are all the same subtitles, and there are no typo subtitles. Deduplication is performed on the second subtitle set to improve the accuracy of obtaining target subtitles.

Exemplarily, the third similarity threshold may be greater than or equal to the aforementioned second similarity threshold.

Exemplarily, the third similarity threshold may be 99% or 100% or the like.

In some embodiments, after the combined pair of text and audio is obtained through the above steps, the audio in the combined pair can be used to check the text to determine whether the combined pair can be used as a training sample for the speech recognition model, thereby improving the utilization The recognition accuracy of the speech recognition model trained by the training samples.

Exemplarily, after obtaining the combined pair of text and audio, it further includes:

Perform speech recognition on the target audio based on the speech recognition model, and obtain the verification text corresponding to the target subtitle;

Determine the confidence of the combination pair based on the target subtitle and its corresponding verification text;

When the confidence of the combination pair satisfies the set confidence threshold, it is determined that the combination pair is a training sample for the speech recognition model.

In this example, the pre-trained speech recognition model is used to perform speech recognition on the target audio, and the verified verification text obtained by the recognition is verified against the target subtitle to obtain the confidence of the combined pair consisting of the target audio and the target subtitle degree, and determining whether to determine the combination pair as a training sample based on the confidence degree can improve the training accuracy of the speech recognition model.

Exemplarily, the confidence of the combination pair can be determined by using the editing distance between the target subtitle and its verification text, the character length of the target subtitle and the character length of the verification text.

Exemplarily, assuming that the result recognized by the speech recognition model ASR is string1, and the subtitle result obtained by text recognition OCR is string2, then the confidence C of the training sample can be expressed as:

C=1-EditDistance(string1,string2)/max(len(string1),len(string2))

Among them, EditDistance(string1, string2) indicates the edit distance between string1 and string2, len(string1) is the length of string1, len(string2) is the length of string2, and the max() function is a function for obtaining the maximum value.

When the consistency between the ASR recognition result and the OCR recognition result is high, that is, the consistency between string1 and string2 is high, the value of the confidence degree C of the training sample is high; otherwise, when the consistency between the two is low, the value of the confidence degree C is low. Therefore, samples with low labeling quality can be filtered out through the confidence C.

Exemplarily, the above training samples can be applied to the training of any speech recognition model. Among them, the target audio is used as the input of the speech recognition model, and the target subtitle is used as the annotation of the target audio, which is compared with the output of the speech recognition model, and then the parameters of the speech recognition model are adjusted according to the comparison result.

In some embodiments, the training samples generated above cannot be directly applied to model training, and there may be some numbers or function symbols in the text in the training samples, which need to be regularized.

Exemplarily, after obtaining the training samples, the above method further includes:

In the case that the target subtitles have target characters, regularization processing is performed on the target subtitles.

In this example, regularization is performed on target subtitles with target characters, and the obtained training samples can be applied to model training.

Exemplarily, the target characters may include numbers, function symbols, formulas, time and date, units and the like. Convert the target character to the corresponding Chinese character. Generally, it can be realized through methods such as dictionary mapping and regular matching.

For example, this piece of gold weighs 324.75 grams, after regularization: this piece of gold weighs 324.75 grams; she was born on August 18, 1986 and her brother was born on March 1, 1995, After regularization: she was born on August 18, 1986 and her brother was born on March 1, 1995; there is a 62% probability of rain tomorrow, and after regularization, it is: 60% tomorrow The probability of rain is two; this is the mobile phone 8618544139121, after regularization: this is the mobile phone 8618544139121.

The following will describe an application example of constructing a training sample set of a speech recognition model based on the combination of OCR recognition and ASR recognition, as follows:

As shown in Figure 5, the construction process of the training sample set of the speech recognition model mainly includes: video key frame extraction, OCR text recognition, watermark and scene non-subtitle text filtering, subtitle switching detection, text error correction and generation, and text regularization And sample annotation verification.

(1) Video key frame extraction

Traverse each frame in the video for text detection, and compare whether the two frames before and after are repeated by the frame difference method. If it is repeated, delete the redundant frame, and finally get the key frame containing the subtitle. The method of judging whether the two frames before and after are repeated is to compare the similarity of the text strings extracted from the two frames of images. The calculation method of the similarity may be an edit distance or the like. The text detection and recognition module for image frames can be implemented with the RCNN deep learning model. In addition, key frames can be extracted quickly based on FPS (frames per second). For example, when the FPS is 24, 3 frames per second can be extracted as key frames, and a key frame can be extracted every 8 frames, so as to get the key frame set. This extraction method is faster. A specific method for extracting key frames may be determined according to characteristics of video data or resource conditions.

(2) OCR text recognition

After the key frame of the video file is obtained, the text information of the text block and the position information of each text block in the key frame can be identified from the key frame by using the OCR technology. In general, the results obtained by OCR are [(text1,(xmin,xmax,ymin,ymax)),(text2,(xmin,xmax,ymin,ymax)),……]. Among them, text1 and text2 are the recognized text information, xmin, xmax represent the minimum abscissa and maximum abscissa of the text block in the key frame, ymin, ymax represent the minimum middle ordinate and maximum ordinate of the text block in the key frame , specifically as shown in Figure 4.

(3) Non-subtitle text filtering for watermark and background

Background information such as station logos or advertisements may exist in some TV dramas and variety shows. For example, "XXTV TV drama channel" in FIG. 4 is a station logo. This contextual information tends to be at a fixed location in the video and exists throughout the duration of the video file. Therefore, by counting the frequency of the coordinate information of all texts in the OCR recognition result, these background contents can be filtered out through the frequency threshold, so as to prevent the information from being mistaken for subtitle information. It should be noted that there will be slight differences in the position coordinate information of the same text recognized by OCR in different frames. For example, in Figure 3, the position coordinates of "study hard and improve every day" recognized in the two frames before and after may be (125, 512, 678, 780) and (126, 513, 679, 781), so their coordinates need to be "normalized". That is, when the coordinate difference of two text blocks in two different frames is smaller than a predetermined difference threshold, they can be regarded as the same coordinate. For example, when the coordinate difference between the text block (x1, y1) and the text block (x2, y2) is ||x1-x2||<a,||y1-y2||<b, then the two text blocks The positions are considered to be the same. Wherein a and b are predetermined thresholds, which can be set according to actual conditions, for example, a is 50 pixel values, and b is 100 pixel values. After filtering out the watermark and background content, sort the remaining text according to the frequency of the text, and determine the coordinate area of the text with the highest frequency as the subtitle area, which can be recorded as (Xmin, Xmax, Ymin, Ymax), If the text is located in the area, or the area where the text is located matches the subtitle area, then the text is determined as the subtitle.

(4) Subtitle switching detection, text error correction and generation

As shown in Figure 6, after obtaining the subtitles of the video, we need to merge the text blocks of the same frame, such as the case of double-line subtitles. At the same time, in the case of redundant key frames, there will be a problem of repeated subtitles, and the repeated subtitles need to be deleted.

As shown in Figure 7, in the subtitle position verification module, in the subtitles obtained by text recognition, there may be problems of matching and adaptation due to multi-line subtitles or insufficient background filtering. in the video produced. As shown in Figure 3A and Figure 3B, it is the scene when the teacher is teaching, there are background subtitles in the video, and there are multiple lines of subtitles. If the subtitle position obtained above is used to verify the current text position, there will be a situation where the subtitle position does not match the text position, that is, the position area where the text is located (Xminc, Xmaxc, Yminc, Ymaxc) is greater than the subtitle position, such as Yminc <Ymin,Ymaxc>Ymax, in this case, if the subtitle is determined only based on the subtitle position, the subtitle content will be lost, but the text information outside the subtitle area cannot be directly regarded as subtitles by default. Therefore, the position area of the current text can be marked, the corresponding audio can be obtained by using the time stamp of the text switching in the current text, and then the audio can be recognized by ASR to obtain the verification text, and the text recognized by the current OCR can be summoned by the verification text. The specific recall method is as follows:

Calculate the coincidence degree m=len(string(ocr)&string(asr))/len(string(ocr) between the OCR recognition text line string(ocr) and the ASR recognition result string(asr);

Among them, len() refers to the function of taking the character length, and string(ocr)&string(asr) refers to the intersection character between the character string(ocr) and the character string(asr);

When the coincidence degree is greater than the set coincidence degree threshold, the OCR-recognized text line is determined as the subtitle content.

As shown in FIG. 6, after all subtitles of the video file are obtained, all subtitles of the same frame are combined. Then, subtitle switching detection shown in FIG. 7 is performed.

This application example uses the similarity of subtitle information between different frames to detect subtitle switching. As shown in Figure 7, traverse the subtitle line in turn, record the start frame frame_s of the current frame number, compare the subtitle similarity between the subtitle of the current frame and the subtitle of the next frame, when the subtitle similarity of the adjacent frame is greater than the preset If the specified threshold (such as 0.9), the subtitle content of the two frames is the same by default, then skip the next frame and continue to compare with the next frame until a different subtitle frame is compared, and record the current frame number as the end frame number of the subtitle frame_e, and so on, until all subtitle lines are traversed. Thus, the time stamps at which subtitle switching occurs in the current subtitle are: frame_s and frame_e.

The similarity scheme mentioned above can be calculated by using the edit distance of the string. For example, the similarity comparison can be performed based on the edit distance of IDS (Ideographic Description Sequence). IDS only characterizes hieroglyphs in a sequence, similar to the sequence of letters in English, and Chinese can be represented by strokes and Chinese character structures. As shown in Figure 8, the IDS representation of poor characters. This representation method is more powerful and robust than pure form and near-character representation.

If the subtitle similarity of adjacent frames is greater than a predetermined threshold but not completely the same during subtitle switching detection, it indicates that there may be an OCR recognition error in the current subtitle. For example, the recognition results of two adjacent frames are "the traffic at the current intersection is heavy, so be careful" and "the traffic at the current intersection is heavy, you should be careful". In this example, OCR recognizes "person" as "in". When OCR performs text recognition, it will give the result of recognizing the text, the location information of the text, and the confidence of the text. Therefore, the confidence in the OCR recognition result can also be used to determine whether the current character has an error and needs to be modified.

Therefore, this application example considers text error correction for erroneous subtitle content. Error correction can be performed using the pre-trained language model BERT in natural language processing. BERT is called a MASK (covered) language model, that is, its main task in the training phase is to predict the elements that are masked. For example, when text is input into the BERT model, mask suspected wrong words: "The current [MASK] traffic is very heavy, so be careful", and then use the BERT model to predict what the masked words are. Through the set of candidate words in the MASK position predicted by BERT, the confidence p of each word, and the IDS edit distance s between the candidate word and the masked word, determine what the final word with the correct MASK position is. For example, the word in the MASK position predicted by BERT is:

1) For "vehicle", the confidence p is 0.8123, and the IDS edit distance s from "entry" is 0.012;

2) The confidence p of "person" is 0.2313, and the edit distance s with the IDS of "in" is 0.9342;

The final scoring rule is that the highest value of Score=p+s is the final candidate word.

In some embodiments, additional restrictive conditions can be added to improve accuracy and avoid erroneous modification. For example, the final shape similarity s must be greater than 0.5, and the Score value must be greater than 1 to modify the original character, otherwise it remains unchanged.

(5) Text regularization and sample label verification

The subtitle information of the finally generated video file is a subtitle file in srt format, which includes time code + subtitle information, as shown in FIG. 9 .

The original audio file can be extracted from the original video through audio and video processing tools such as ffmpeg, combined with the subtitle file, the audio can be segmented, and the audio and subtitle can be matched to form a <text, audio> text-audio pair, which is the speech recognition training Desired training samples of text and speech.

However, the data generated by the above process cannot be directly used for the training of the ASR model, and the text needs to be regularized. Text regularization refers to the expression of Elibe numbers, time and date, units, etc. that appear in the text with Chinese characters. Generally, it can be realized by dictionary mapping and regular matching, as shown below:

1) This piece of gold weighs 324.75 grams --> regularization --> this piece of gold weighs 324.75 grams;

2) She was born on August 18, 1986 and her brother was born on March 1, 1995 --> regularization --> she was born on August 18, 1986. Her brother was born on March 1995 the first day of the month;

3) There is a 62% probability of rain tomorrow --> regularization --> there is a 62% probability of rain tomorrow;

4) This is the mobile phone 8618544139121 --> regularization --> this is the mobile phone 8618544139121.

In addition, in order to ensure the quality of the training samples, the pre-trained ASR model can be used to verify the OCR-marked text in the training samples. Assuming that the audio recognition result obtained by ASR is string1, and the subtitle result recognized by OCR is string2, the confidence level of the data sample label can be expressed as:

C=1-EditDistance(string1,string2)/max(len(string1),len(string2))

When the consistency between the ASR recognition result and the OCR recognition result is high, the consistency between string1 and string2 is high, and the confidence degree C is high. On the contrary, when the consistency between the ASR recognition result and the OCR recognition result is low, the confidence degree C is low. Therefore, the training samples with low labeling quality can be filtered out through the confidence C value.

FIG. 10 is a block diagram of an audio and text combining device according to an embodiment of the present disclosure.

As shown in FIG. 10, an embodiment of the present disclosure provides an audio and text combination device, including:

Text recognition module 110, is used for carrying out text recognition to the key frame in video file, obtains the first subtitle set;

A subtitle switching detection module 120, configured to perform subtitle switching detection on the first subtitle set, to obtain the target subtitle and the timestamp when the subtitle switch occurs in the target subtitle;

The target audio interception module 130 is used to intercept the audio from the audio file of the video file based on the timestamp to obtain the target audio;

The audio-text combination module 140 is configured to construct a combination pair of text and audio based on the target subtitle and the target audio.

In some possible implementations, the above-mentioned text recognition module 110 may include:

A candidate subtitle recognition unit is used to perform text recognition on key frames in the video file to obtain candidate subtitle text;

A verification audio acquisition unit, configured to determine the verification audio corresponding to the candidate subtitle text in the video file;

A subtitle verification unit, configured to determine that the candidate subtitle text is the subtitle text of the video file when the relationship between the candidate subtitle text and the verification audio meets a set condition;

The first subtitle set acquiring unit is configured to acquire the subtitle text of the video file to obtain the first subtitle set.

Exemplarily, the subtitle checking unit is specifically used for:

performing speech recognition on the verification audio to obtain the verification text;

If the coincidence degree between the candidate subtitle text and the verification text is greater than a set coincidence degree threshold, it is determined that the candidate subtitle text is the subtitle text of the video file.

In some possible implementation manners, the candidate subtitle identification unit is specifically configured to:

performing text recognition on each key frame in the video file to obtain the first text set;

determining non-subtitle text based on the first text set;

filtering the non-subtitle text in the first text set to obtain a second text set;

Based on the second text set, candidate subtitle texts are determined.

In some possible implementation manners, the determining the non-subtitle text based on the first text set includes:

Based on the position information of each text in the first text block set, determine the frequency of occurrence of the text with the same position;

If the frequency of occurrence is greater than a set frequency threshold, the text with the same position is determined as non-subtitle text.

In some possible implementation manners, the determining candidate subtitle text based on the second text set includes:

determining a first subtitle area based on the location information of each text in the second text set;

For each text in the second text set, if the matching degree between the position information of the text and the first subtitle area is less than a set matching threshold, determine that the text is a candidate subtitle text.

In some possible implementation manners, the above-mentioned determination of candidate subtitle texts based on the second text set further includes:

For each text in the second text set, when the matching degree of the position information of the text and the first subtitle area is greater than a set matching threshold, it is determined that the text is the subtitle of the video file text.

In some possible implementation manners, the above candidate subtitle identification unit is specifically configured to:

performing text recognition on each key frame in the video file to obtain a third text set;

determining a second subtitle area based on the position information of each text in the third text set;

For each text in the third text set, when the matching degree between the position information of the text and the second subtitle area is greater than a set matching threshold, determine that the text is the subtitle of the video file text;

Obtain the subtitle text of the video text to obtain the first subtitle set.

In some possible implementations, the subtitle switching detection module 120 includes:

The subtitle division unit is used to determine the timestamp of subtitle switching based on the similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, and use the timestamp to update the first subtitle set performing division to obtain multiple second subtitle sets;

a subtitle deduplication unit, configured to deduplicate the second subtitle set to obtain target subtitles;

A timestamp determining unit, configured to determine, based on the timestamp of the second subtitle set, the timestamp at which subtitle switching occurs for the target subtitle.

In some possible implementations, the subtitle switching detection module 120 further includes:

The subtitle error correction unit is used to ensure that the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than the first similarity threshold and less than the second subtitle text before deduplicating the second subtitle set. In the case of a similarity threshold, text error correction is performed on the first subtitle text and the second subtitle text.

In some possible implementation manners, the subtitle error correction unit is specifically configured to:

In the first subtitle text and the second subtitle text, determine words with the same position but different words as masked words;

In the first subtitle text and the second subtitle text, masking the masked words to obtain a third subtitle text;

Determine target prediction words based on the third subtitle text and the masked words;

Based on the target predicted word, the masked words in the first subtitle text and the second subtitle text are corrected.

In some possible implementation manners, the above subtitle deduplication unit is specifically used for:

When the similarity between any two subtitle texts in the second subtitle set is greater than a third similarity threshold, deduplication is performed on the second subtitle set to obtain a target subtitle.

In some possible implementations, the above device also includes:

A verification text acquisition module, configured to perform speech recognition on the target audio based on the speech recognition model, to obtain the verification text corresponding to the target subtitle;

A confidence degree determination module, configured to determine the confidence degree of the combination pair based on the target subtitle and its corresponding verification text;

The training sample determination module is configured to determine the combination pair as a training sample for the speech recognition model when the confidence of the combination pair satisfies a set confidence threshold.

In some possible implementations, the above device also includes:

A regularization processing module, configured to perform regularization processing on the target subtitle when there are target characters in the target subtitle.

For functions of each unit, module, or submodule in each device in the embodiments of the present disclosure, reference may be made to the corresponding descriptions in the foregoing method embodiments, and details are not repeated here.

According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 11 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 11 , the electronic device 800 includes a computing unit 801 that can be executed according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803 Various appropriate actions and treatments. In the RAM 803, various programs and data necessary for the operation of the electronic device 800 can also be stored. The computing unit 801 , ROM 802 , and RAM 803 are connected to each other through a bus 804 . An input output (I/O) interface 805 is also connected to the bus 804 .

Multiple components in the electronic device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the electronic device 800 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 801 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes various methods and processes described above, such as the method of combining audio and text. For example, in some embodiments, the audio and text combining method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 800 via the ROM 102 and/or the communication unit 809 . When the computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the audio and text combining method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other suitable way (for example, by means of firmware) to execute the method of combining audio and text.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to processors or controllers of general-purpose computers, special-purpose computers, or other programmable ambient light adjustment devices, so that when the program codes are executed by the processor or controller, the functions specified in the flowchart and/or block diagrams may be implemented. / operation is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Input from the user may be received through acoustic input, voice input, or tactile input.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (18)

1. An audio and text combining method, comprising:

performing text recognition on key frames in a video file to obtain a first subtitle set;

performing subtitle switching detection on the first subtitle set to obtain a target subtitle and a timestamp of subtitle switching of the target subtitle;

intercepting audio from an audio file of the video file based on the timestamp to obtain target audio;

and constructing a combined pair of text and audio based on the target caption and the target audio.

2. The method of claim 1, wherein the text recognition of the key frames in the video file to obtain the first caption set comprises:

performing text recognition on key frames in a video file to obtain candidate subtitle texts;

in the video file, determining a check audio corresponding to the candidate subtitle text;

determining the candidate subtitle text as the subtitle text of the video file under the condition that the relation between the candidate subtitle text and the check audio accords with a set condition;

and acquiring a subtitle text of the video file to obtain a first subtitle set.

3. The method according to claim 2, wherein the determining that the candidate subtitle text is the subtitle text of the video file in the case that the relationship between the candidate subtitle text and the verification audio meets a set condition comprises:

performing voice recognition on the verification audio to obtain a verification text;

and under the condition that the coincidence degree between the candidate subtitle text and the check text is greater than a set coincidence degree threshold value, determining that the candidate subtitle text is the subtitle text of the video file.

4. The method of claim 2, wherein the performing text recognition on the key frames in the video file to obtain candidate subtitle texts comprises:

performing text recognition on key frames in a video file to obtain a first text set;

determining non-subtitle text based on the first text set;

filtering the non-subtitle texts in the first text set to obtain a second text set;

and determining candidate subtitle texts based on the second text set.

5. The method of claim 4, wherein determining non-caption text based on the first set of text comprises:

determining the occurrence frequency of texts with the same positions based on the position information of each text in the first text block set;

and determining the text with the same position as the non-subtitle text when the frequency of occurrence is greater than a set frequency threshold.

6. The method of claim 4, wherein determining candidate subtitle text based on the second set of text comprises:

determining a first subtitle area based on the position information of the text in the second text set;

and for the text in the second text set, determining the text as a candidate subtitle text when the matching degree of the position information of the text and the first subtitle region is smaller than a set matching threshold.

7. The method of claim 6, further comprising:

and for the text in the second text set, determining that the text is the subtitle text of the video file when the matching degree of the position information of the text and the first subtitle region is greater than a set matching threshold.

8. The method of claim 1, wherein performing text recognition on the key frames of the video file to obtain the first subtitle set comprises:

performing text recognition on key frames in the video file to obtain a third text set;

determining a second subtitle area based on position information of the text in the third text set;

for the text in the third text set, determining that the text is the subtitle text of the video file when the matching degree of the position information of the text and the second subtitle region is greater than a set matching threshold;

and acquiring a subtitle text of the video text to obtain a first subtitle set.

9. The method of claim 1, wherein the performing caption switching detection on the first caption set to obtain a target caption and a timestamp of the target caption when caption switching occurs comprises:

determining a timestamp for switching subtitles based on the similarity between any two subtitle texts belonging to adjacent key frames in the first subtitle set, and dividing the first subtitle set by using the timestamp to obtain a plurality of second subtitle sets;

removing the duplicate of the second caption set to obtain a target caption;

and determining the timestamp of subtitle switching of the target subtitle based on the timestamp of the second subtitle set.

10. The method of claim 9, wherein prior to de-duplicating the second subtitle set, the method further comprises:

and under the condition that the similarity between the first subtitle text and the second subtitle text in the second subtitle set is greater than a first similarity threshold and smaller than a second similarity threshold, performing text error correction on the first subtitle text and the second subtitle text.

11. The method of claim 10, wherein the text error correcting the first subtitle text and the second subtitle text comprises:

determining words with the same position and different words in the first subtitle text and the second subtitle text as shielding words;

shielding the shielding words in the first subtitle text and the second subtitle text to obtain a third subtitle text;

determining a target predicted word based on the third caption text and the shielding word;

and modifying the shielding words in the first subtitle text and the second subtitle text based on the target predicted words.

12. The method of claim 9, wherein the de-duplicating the second caption set to obtain the target caption comprises:

and under the condition that the similarity between any two subtitle texts in the second subtitle set is greater than a third similarity threshold, carrying out duplicate removal on the second subtitle set to obtain a target subtitle.

13. The method of claim 1, further comprising:

performing voice recognition on the target audio based on a voice recognition model to obtain a check text corresponding to the target subtitle;

determining the confidence of the combined pair based on the target caption and the corresponding check text thereof;

and determining the combined pair as a training sample of the speech recognition model under the condition that the confidence coefficient of the combined pair meets a set confidence coefficient threshold value.

14. The method of claim 1, further comprising:

and under the condition that target characters exist in the target subtitles, regularizing the target subtitles.

15. An audio and text combining apparatus, comprising:

the text recognition module is used for performing text recognition on key frames in the video file to obtain a first subtitle set;

the subtitle switching detection module is used for carrying out subtitle switching detection on the first subtitle set to obtain a target subtitle and a timestamp of subtitle switching of the target subtitle;

the target audio intercepting module is used for intercepting audio from the audio file of the video file based on the timestamp to obtain target audio;

and the audio text combination module is used for constructing a text and audio combination pair based on the target subtitle and the target audio.

16. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

17. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-14.

18. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-14.

Images