JP7481894B2 - Speech text generation device, speech text generation program, and speech text generation method - Google Patents

Description

The present invention relates to a speech text generation device, a speech text generation program, and a speech text generation method.

In recent years, in the fields of voice synthesis and voice recognition, a method of performing voice synthesis and voice recognition using a deep neural network (DNN) has become common.
For example, Patent Document 1 discloses a voice synthesis method that generates voice data from text data using a statistical model configured with a DNN.
Furthermore, a speech recognition method for generating text data from speech data using an acoustic model configured with a DNN is disclosed in, for example, Japanese Patent Laid-Open No. 2003-233996.
Such a method using DNN requires a huge amount of training data to train the DNN model.
Conventionally, as a method for generating this learning data, a method is disclosed in Patent Document 3 and the like in which learning data is generated by associating audio data and text data corresponding to a time from audio data and subtitle data (text data) of a broadcast program.

JP 2019-219590 A JP 2019-020597 A Patent No. 6426971

When extracting audio data and closed caption data, which is text data, in association with each other from broadcast data on which closed caption data, such as a broadcast program, is superimposed, using a conventional method, the following problems arise.
When a broadcast program is live, the subtitle creator adds subtitles by keyboard after listening to the transmitted audio, so the subtitles are broadcast with a delay from the actual audio. Therefore, in the conventional method, a time lag occurs between the audio data and the subtitle data, and the learning data cannot be generated correctly.
Furthermore, in live broadcasts, the addition of subtitles requires manual intervention, and the time lag between the audio data and the subtitle data is not constant, making it difficult to synchronize and associate the audio data and the subtitle data.

The present invention aims to provide a speech text generation device, a speech text generation program, and a speech text generation method that are capable of generating speech data and text data for each speech section from speech data containing multiple speech sounds and corresponding text data.

In order to achieve the above object, the speech text generation device according to the present invention is configured to include a speech segment detection means, a speech recognition means, a matching means, a context information generation means, and a conversion means.

In such a configuration, the speech text generation device detects, by the speech segment detection means, segment positions of section speech data for each utterance from speech data consisting of a plurality of utterances, based on acoustic features such as power.
Then, the speech text generation device performs speech recognition on each section speech data of the speech section by using the speech recognition means.
Then, the speech text generation device estimates section text data corresponding to the time of the section speech data by using a matching means to match the recognition result of the speech recognition means with text data, which is the spoken content of the speech data, using a matching method such as DP matching.

Furthermore, the speech text generation device generates, from the section text data, context information for each phoneme, including at least information on the phoneme and accent phrase information indicating characteristics of an accent phrase including the phoneme and an accent phrase adjacent to the accent phrase, by using the context information generation means. This context information makes it possible to recognize the accent state of each phoneme.
The speech text generation device then converts the context information of the phoneme string into second section text data including characters representing the reading order of the phonemes and predetermined characters representing prosody indicating the state of accent, by using the conversion means, thereby making it possible to generate section text data corresponding to the time of the section speech data uttered by the speaker, that is, the second section text data.

In order to achieve the above object, a speech text generation program according to the present invention can be realized as a program for causing a computer to function as each of the means described above.
In order to achieve the above object, the speech text generation method according to the present invention can be realized as a procedure including the operations of the above-mentioned means as steps.

The present invention provides the following excellent effects.
According to the present invention, even if there is a time lag between voice data consisting of multiple utterances and the text data corresponding to that voice data, it is possible to extract voice data and text data for each utterance in association with each other.

1 is a block diagram showing a configuration of a training data generation system including a speech text generation device according to an embodiment of a reference example . FIG. 13 is a diagram showing an example of a selection screen for selecting a file of subtitled data in an upload terminal. 13 is a diagram showing an example of an editing screen for correcting audio delimiter positions and text data on an editing terminal. FIG. 13 is a flowchart showing the operation of a training data generation system according to an embodiment of a reference example . 1 is a block diagram showing a configuration of a training data generation system including a speech text generation device according to an embodiment of the present invention. 1 is an explanatory diagram for explaining an example of PLP data including pronunciation kana and prosodic symbols generated by a speech text generation device; FIG. 11 is an explanatory diagram for explaining an example of prosodic symbols. FIG. 13 is a diagram (part 1) showing the characteristics of each label of context information. FIG. 2 is a diagram (part 2) showing the characteristics of each label of context information. FIG. 13 is a diagram illustrating an example of a format of context information. FIG. 11 is an explanatory diagram for explaining a condition for inserting a prosodic symbol. FIG. 11 is an explanatory diagram for explaining a flow of generating PLP data from context information. 11 is a diagram showing an example of an editing screen for correcting audio delimiter positions and text data (PLP data) on an editing terminal. FIG. 13 is a diagram showing an example of an editing screen for correcting audio delimiter positions and text data (kana-kanji mixed text, PLP data) on an editing terminal. FIG. 4 is a flowchart showing the operation of the training data generation system according to the embodiment of the present invention.

Hereinafter, reference examples and embodiments of the present invention will be described with reference to the drawings.
<Embodiments of Reference Examples >
<Configuration of the learning data generation system>
First, with reference to FIG. 1, a configuration of a training data generation system 100 according to an embodiment of the reference example will be described.

The training data generation system 100 generates, as training data, speech data in units of utterances for training a deep neural network (DNN) model used for voice synthesis or voice recognition, and text data corresponding to that speech data.
The training data generation system 100 includes a subtitled data storage device 1, an upload terminal 2, a speech text generation device 3, and an editing terminal 4.

[Subtitled data storage device]
The subtitled data storage device 1 stores subtitled data including audio data consisting of a plurality of spoken voices and subtitle data corresponding to the audio data. The subtitled data is, for example, video and audio content in a data format such as XDCAM (registered trademark). Note that the subtitled data may be data that does not include video data as long as it includes at least audio data and the corresponding subtitle data.
In the subtitled data storage device 1, a plurality of subtitled data are stored in advance as one file.

[Uploading terminal]
The upload terminal 2 is a client terminal that separates audio data and subtitle data (text data) from the subtitled data stored in the subtitled data storage device 1 or from the subtitled data currently being broadcast (broadcast data), and transmits the separated audio data and subtitle data to the speech text generation device 3.
The upload terminal 2 comprises a file selection means 20, a file separation means 21, a broadcast data receiving means 22, and a broadcast data separation means 23.

The file selection means 20 selects a file for generating learning data from among the multiple subtitled data files stored in the subtitled data storage device 1 .
For example, the upload terminal 2 causes the file selection means 20 to display a selection screen G1 for selecting a file as shown in FIG. 2 on a display device (not shown), and the operator selects a file by operating an input means (not shown) such as a mouse.

The selection screen G1 shown in FIG. 2 shows an example in which columns for execution g1, identification name g2, date and time g3, file path g4, channel g5, and status g6, as well as a start button B are displayed.
The execution g1 column is a check column for selecting a file to be selected. Here, when the execution g1 column is selected, the file selection means 20 displays a check mark to indicate that the file has been selected.
The identification name g2 field displays a name for identifying the subtitled data, for example, a file name of the subtitled data.
The date and time g3 is a field for displaying time information of the subtitled data. This time information is the date and time when the subtitled data was recorded or filmed, or the date and time when the subtitled data was stored in the subtitled data storage device 1.
The file path g4 field displays the file path of the subtitled data storage device 1 that stores the subtitled data.

The channel g5 field is a field for specifying a channel number. For example, if the subtitled data is XDCAM, the audio channel to be extracted is selected from a maximum of eight channels.
The status g6 column is a column that displays the upload status of the selected subtitled data. For example, here, it shows a state where the file has only been selected but has not yet been uploaded (transmitted). This status g6 column is updated to "transmission completed" when the voice data and subtitle data are separated by the file separation means 21, which will be described later, and uploaded to the speech voice text generation device 3.
The start button B is a button for instructing uploading of a selected file. After a file has been selected, the upload terminal 2 starts uploading the file when the start button B is pressed using an input means such as a mouse.

Returning to FIG. 1, the description of the configuration of the upload terminal 2 will be continued.
The file selection means 20 reads out the selected subtitled data from the subtitled data storage device 1 and outputs it to the file separation means 21 .

The file separating means (separating means) 21 separates the subtitled data selected by the file selecting means 20 into audio data and subtitle data.
For example, when the subtitled data is XDCAM video content, the video data, audio data, and subtitle data are stored in the content in the MXF (Material eXchange Format) format.
Therefore, the file separation means 21 extracts the audio stream from the MXF subtitled data and converts it into a WAV file, thereby separating the audio data.
The subtitle data is stored in the content as an ARIB (Association of Radio Industries and Businesses) subtitle file in MXF format.
Therefore, the file separation means 21 extracts the ARIB subtitle file from the MXF subtitled data, and converts the ARIB subtitles into a character code (for example, UTF-8) to separate the subtitle data as text data.

The file separation means 21 uploads the separated voice data and text data to the speech text generation device 3 .
Here, the file separation means 21 transmits the voice data and the text data in association with each other to the speech text generation device 3 via the networks N and _N1 by using a communication means (not shown).
After uploading the voice data and text data to the speech voice text generator 3, the file separator 21 updates the status g6 field of the selection screen G1 shown in FIG. 2 to "Transmission Completed".
This allows the operator to check the upload status of the selected file.

The broadcast data receiving means 22 receives subtitled data (broadcast data) being broadcast by digital broadcasting, and demodulates the data into stream data (transport stream (TS)).
For example, when a channel broadcasting subtitled broadcast data is specified from outside, the broadcast data receiving means 22 analyzes the PSI/SI (Program Specific Information/Service Information) in the demodulated stream data and extracts the stream data corresponding to the specified channel.
The broadcast data receiving means 22 outputs the extracted stream data to the broadcast data separating means 23 .

The broadcast data separating means (separating means) 23 separates the stream data received by the broadcast data receiving means 22 into audio data and subtitle data (text data).
The broadcast data separation means 23 extracts audio data multiplexed into the stream data and subtitle data, which is text data multiplexed into the stream data as closed captions.
The broadcast data separation means 23 uploads the separated voice data and text data to the speech text generation device 3 .
Here, the broadcast data separation means 23 transmits the voice data and the text data in association with each other to the speech voice text generation device 3 via the networks N and _N1 by using a communication means (not shown).

Although the configuration of the upload terminal 2 has been described above, the upload terminal 2 is not limited to this configuration. For example, the upload terminal 2 may be configured to omit the broadcast data receiving means 22 and the broadcast data separating means 23, and separate the audio data and text data from the subtitled data stored in the subtitled data storage device 1, and transmit them to the speech voice text generation device 3. Also, for example, the upload terminal 2 may be configured to omit the file selection means 20 and the file separating means 21, and separate the audio data and text data from the broadcast data currently being broadcast, and transmit them to the speech voice text generation device 3.

[Speech to text generator]
The speech text generation device 3 is a server that generates speech data of an utterance section (section speech data) and text data corresponding to the speech data (section text data) as learning data from speech data consisting of multiple utterances and text data corresponding to the speech data.
The speech text generation device 3 includes a speech text storage means 30 , a speech segment detection means 31 , a speech recognition means 32 , and a matching means 33 .

The voice text storage means 30 stores voice data consisting of a plurality of spoken voices in association with text data corresponding to the voice data. The voice text storage means 30 can be configured with a general storage medium such as a hard disk.
The voice data and text data stored in the voice text storage means 30 are data that are received and stored by communication means (not shown) via the networks N and _N1 and uploaded from the upload terminal 2.

The voice segment detection means 31 detects segment positions of voice data (section voice data) for each utterance from voice data consisting of a plurality of utterances.
The voice segment detection means 31 detects speech segments from the voice data stored in the voice text storage means 30, and detects positions between speech segments (for example, intermediate positions) as segment positions of the voice data.
A general method may be used as the method for detecting the speech section in the speech segment detection means 31. For example, the speech segment detection means 31 extracts power (power spectrum) which is an acoustic feature from the speech data, and if the power is greater than a predetermined threshold, it determines the relevant time section as a speech section and the rest as a non-speech section.
The voice segment detection means 31 outputs the voice data and the segment positions of the voice data to the voice recognition means 32 and the matching means 33 .

The voice recognition means 32 performs voice recognition for each piece of voice data (section voice data) divided by the delimiter positions detected by the voice delimiter detection means 31 .
A general method may be used as the voice recognition method in the voice recognition means 32. The voice recognition means 32 performs voice recognition of the voice data using a language model, an acoustic model, and a pronunciation dictionary, which are not shown.
The voice recognition means 32 outputs the recognition result (a mixture of kanji and kana) for each section of voice data to the matching means 33 .

The matching means 33 matches the result of the voice recognition performed by the voice recognition means 32 with text data corresponding to the voice data stored in the voice text storage means 30 .
The matching means 33 estimates text data (section text data) corresponding to the recognition result by matching the recognition result with the text data on a word or character basis, for example, by a dynamic programming matching method (DP matching). At this time, the matching means 33 calculates a matching error rate (MER) including recognition errors, symbol insertions, and rewritings between the recognition result and the estimated section text data as a degree of similarity.
The matching means 33 regards section text data having a mismatch rate less than a predetermined threshold as text data corresponding to the audio data separated at the separation positions (section audio data).
Then, the matching means 33 associates the section voice data separated at the separation positions with the matched section text data.

The matching means 33 transmits the associated section voice data and section text data to the editing terminal 4 via the networks N and _N2 by means of communication means (not shown).
The matching means 33 does not transmit section text data whose mismatch rate with the recognition result is less than a predetermined threshold together with the corresponding section voice data to the editing terminal 4. Alternatively, the matching means 33 may transmit the section text data as NULL data together with the section voice data to the editing terminal 4.

With the above-described configuration, the speech text generation device 3 can generate section speech data and section text data associated with each utterance as learning data from the speech data and text data. At this time, the speech text generation device 3 can associate the section speech data, which is the speech included in the speech data, with the section text data corresponding to the subtitle data regardless of time lag.
The speech text generation device 3 can operate a computer (not shown) using a speech text generation program for causing the computer to function as each of the above-mentioned means.

[Editing terminal]
The editing terminal 4 is a client terminal that corrects the voice data (section voice data) and the text data (section text data) for each speech section that are associated with each other by the speech text generation device 3 .
The editing terminal 4 includes a learning data storage means 40 and a correction means 41 .

The training data storage means 40 stores, in association with each other, section voice data and section text data for each speech section, which are the training data generated by the speech text generation device 3. The training data storage means 40 can be configured with a general storage medium such as a hard disk.
The section voice data and section text data stored in the learning data storage means 40 are data received and stored from the speech voice text generation device 3 via the networks N and _N2 by communication means (not shown).

The correction means 41 corrects the learning data (section voice data and section text data) in response to an operation by an operator.
The modifying means 41 displays an editing screen G2 as shown in FIG. 3, and modifies the section voice data and the section text data in response to an operation by the operator.

FIG. 3 shows an example in which the editing screen G2 is configured with a delimiter position correction screen g10 for correcting the delimiter positions of the section voice data, and a text correction screen g11 for correcting the section text data.
The delimiter position correction screen g10 is a screen that displays the audio waveform w of the section audio data to be corrected in chronological order along with the audio waveforms wf, wb of the preceding and following section audio data, and also displays the delimiter positions pf, pb before and after the section audio data to be corrected.
The delimiter position correction screen g10 has an interface for correcting the delimiter positions pf and pb by the operator operating a mouse or the like.
The break position correction screen g10 further includes a play button b1, a stop button b2, a pause button b3, a 10 second back button b4, and a 10 second forward button b5, and has an interface that accepts instructions from the operator to play back audio data from a desired position.

The text correction screen g11 is a screen that displays the section text data to be corrected.
The text correction screen g11 has an interface for editing text data by operating a keyboard or the like.
In addition, the editing screen G2 has a back button b6 that switches the object to be corrected to the text of the previous section (section audio data, section text data), a forward button b7 that saves the correction content and advances the object to be corrected to the next section, and a forward button b8 that advances to the next section without saving the correction content or without making any corrections, and has an interface that allows the operator to switch the object to be corrected as desired.

Returning to FIG. 1, the configuration of the editing terminal 4 will be further explained.
The correcting means 41 includes a voice segment correcting means 410 and a text correcting means 411 .

The voice segment correction means 410 corrects the segment positions of the section voice data stored in the learning data storage means 40 .
The audio segment correction means 410 displays the audio waveform w of the section audio data to be corrected in chronological order together with the audio waveforms wf, wb of the preceding and following section audio data on the segment position correction screen g10 of the editing screen G2 shown in FIG.
Moreover, the audio segment correction means 410 displays the segment positions pf and pb of the audio data segment to be corrected.

The audio segment correction means 410 plays or stops audio data from a position desired by the operator in response to an instruction from the play button b1, the stop button b2, the pause button b3, the 10-second back button b4, or the 10-second forward button b5. This allows the operator to determine the optimal segment position of the audio data.
The audio segmentation correction means 410 corrects the segmentation positions pf and pb by the operator dragging the lines of the segmentation positions pf and pb to the left and right with, for example, a mouse.

When the audio segment correction means 410 corrects the previous segment position pf to be moved backward or the next segment position pb to be moved forward, it only has to delete the audio waveform at the specified position in the audio waveform w of the section audio data to be corrected. When the audio segment correction means 410 corrects the previous segment position pf to be further forward or the next segment position pb to be further backward, it only has to add a part of the audio waveform of the previous or next section audio data to the audio waveform w of the section audio data to be corrected.

The text correcting means 411 corrects the section text data stored in the learning data storage means 40 .
The text correction means 411 displays the section text data to be corrected on the text correction screen g11 of the editing screen G2 shown in FIG.
Then, the text correction means 411 corrects the section text data by general text editing in response to the operator's operation of the keyboard or the like.
The correction means 41 changes the object of correction to the previous or next in the chronological order when the back button b6 or the forward button b8 on the editing screen G2 shown in FIG. 3 is pressed by an input means such as a mouse.
Furthermore, when a forward button b7 is pressed by input means such as a mouse, the correcting means 41 updates the data in the learning data storage means 40 with the corrected section voice data and section text data.

As described above, the training data generation system 100 can generate training data from subtitled data (broadcast data), in the form of utterance-based voice data (section voice data) for training a DNN model used for voice synthesis or voice recognition, and text data corresponding to that voice data (section text data).
The editing terminal 4 is not necessarily a required component of the training data generation system 100. However, it is preferable to provide the editing terminal 4 in terms of improving the accuracy of the training data.
Furthermore, the training data generation system 100 may be configured as a speech text generation device that integrates the upload terminal 2, the speech text generation device 3, and the editing terminal 4.

<Operation of the learning data generation system>
Next, the operation of the training data generation system 100 (a method for generating spoken voice text) according to the embodiment of the reference example will be described with reference to FIG. 4 (for the configuration, refer to FIG. 1 as appropriate).
It is assumed that the subtitled data storage device 1 stores subtitled data in advance, the subtitled data including audio data consisting of a plurality of spoken voices and subtitle data corresponding to the audio data.

In step S1, the upload terminal 2 acquires subtitled data. Here, the upload terminal 2 acquires subtitled data selected by the operator from the subtitled data storage device 1 using the file selection means 20. Alternatively, the upload terminal 2 receives broadcast data using the broadcast data receiving means 22 and extracts stream data corresponding to the specified channel.

In step S2, the upload terminal 2 separates the audio data and the text data (subtitle data) from the subtitled data.
In step S3, the upload terminal 2 uploads the separated voice data and text data to the speech text generator 3.

In step S4, the speech text generator 3 stores the speech data and text data uploaded in step S3 in the speech text storage means 30 in association with each other.
In step S5, the speech text generation device 3 detects, by the speech segmentation detection means 31, segmentation positions of the speech data for each utterance in the speech data consisting of a plurality of utterances.
In step S6, the speech text generator 3 performs speech recognition for each section of speech data, which is speech data divided by the delimiter positions detected in step S5, by the speech recognition means 32. This generates a speech recognition result corresponding to the speech data in units of utterances.

In step S7, the speech text generation device 3 estimates text data (section text data) corresponding to the recognition result by using the matching means 33 to match the recognition result of the section speech data generated in step S6 with text data corresponding to the speech data consisting of multiple speech sounds.
In step S8, the speech text generation device 3 transmits the generated training data (section speech data, section text data) to the editing terminal 4, and the editing terminal 4 stores the section speech data and the section text data in association with each other in the training data storage means 40.

In step S9, the editing terminal 4 uses the modifying means 41 to modify the delimiter positions of the section voice data and the character strings of the section text data as required at the discretion of the operator.
Here, the editing terminal 4 uses the voice segment correction means 410 to correct segment positions of the section voice data, and uses the text correction means 411 to correct the section text data.
Through the above operations, the training data generation system 100 can generate training data for training a DNN model used for speech synthesis or speech recognition.

<Embodiments of the present invention >
<Configuration of the learning data generation system>
Next, a configuration of a training data generation system 100B according to an embodiment of the present invention will be described with reference to FIG.
In the speech synthesis method described in the following reference document, the DNN used for speech synthesis is trained using speech data and the corresponding kana pronunciation and prosodic symbols as training data.
(References) Kiyoshi Kurihara, Nobumasa Kiyoyama, Tadashi Kumano, Atsushi Imai, "Study on Japanese End-to-End Speech Synthesis Method Using Reading Kana and Prosodic Symbols as Input," Acoustical Society of Japan Autumn Meeting, 1-4-1, Sep. 2018.
This reference document describes that the quality of speech synthesis results is improved by using pronunciation kana and prosodic symbols as training data rather than text data containing a mixture of kanji and kana or only katakana.

The training data generation system 100B shown in FIG. 5 is a system that makes it possible to generate training data for the techniques described in the reference documents.
The training data generation system 100B generates utterance-unit speech data and pronunciation and prosodic symbols corresponding to the speech data as training data for training a deep neural network (DNN) model used for speech synthesis or speech recognition.

Here, the pronunciation and prosodic symbols generated by the training data generation system 100B will be described with reference to FIG. 6 and FIG.
FIG. 6 shows an example in which the reading kana and prosodic symbols corresponding to "Konichiwa Shogo no News Desu" (a mixed kanji and kana sentence) are written.
Here, "Konnichiwa Shogo no Niyusu desu" is the pronunciation, and the symbols added in the middle or at the end of the pronunciation are prosodic symbols.
The pronunciation is as follows:
Any characters that represent readings may be used, and other than katakana, hiragana, phonetic symbols, pronunciation symbols, roman letters, etc. may be used.
The prosodic symbols are predetermined characters that represent prosody, and are symbols that indicate the position or state of accents, divisions of phrases and sentences, intonation at the end of a sentence, pauses, and the like.

Figure 7 shows examples of prosodic symbols. To specify the position of the accent, the prosodic symbol "" representing an accent rise and the prosodic symbol "&" representing an accent fall are used. To specify the division of a phrase or phrase, the prosodic symbol "#" representing the division of an accent phrase is used. To specify the intonation at the end of a sentence, the prosodic symbol "(" representing the normal end of a sentence and the prosodic symbol "?" representing the end of a question are used. To specify a pause, the prosodic symbol "_" is used. Note that these prosodic symbols are only examples, and other symbols may be used. Also, in these examples, the prosodic symbol is represented by one character, but it may be represented by two or more characters. Also, in addition to the prosodic symbols shown in Figure 7, prosodic symbols of other prosody may be used.

Returning to FIG.
The training data generation system 100B includes a subtitled data storage device 1, an upload terminal 2, a speech voice text generation device 3B, and an editing terminal 4.
The subtitled data storage device 1, the upload terminal 2, and the editing terminal 4 have the same configuration as those described in FIG. 1, and therefore a description thereof will be omitted.

[Speech to text generator]
The speech text generation device 3B is a server that generates speech data of an utterance section (section speech data) and reading kana and prosodic symbols, which are text data corresponding to the speech data, as learning data from speech data consisting of a plurality of utterances and text data corresponding to the speech data. Note that the reading kana and prosodic symbols may be referred to as PLP (Symbols of phoneme and linguistic phonological features) data.

The spoken voice text generation device 3B includes a voice text storage means 30, a voice segment detection means 31, a voice recognition means 32, a matching means 33, a context information generation means 34, and a conversion means 35.
The voice text storage means 30, the voice segment detection means 31, the voice recognition means 32 and the matching means 33 have the same configuration as those described in Fig. 1, and therefore the description thereof will be omitted. Note that, in this embodiment, the matching means 33 outputs section text data to the context information generation means 34 and outputs section voice data to the conversion means 35.

The context information generating means 34 generates context information (context label data) from the section text data (a mixture of kanji and kana text) associated with the section voice data by the matching means 33 .
Context information is information that represents information (context) for each phoneme, which includes at least phoneme information and accent phrase information that indicates characteristics related to the accent phrase in which the phoneme is included and the accent phrases adjacent to the accent phrase, for each predetermined index (label).

8 and 9 show the characteristics of each label of the context information. n indicates the order of the phonemes when the first phoneme is numbered 1. The labels p _n , a _n to k _n indicate the characteristics when the nth phoneme is the current position.
p _n represents a sequence of phonemes centered on the current (nth) phoneme. p _n,1 represents the phoneme two steps back (first preceding phoneme), p _n,2 represents the phoneme one step back (preceding phoneme), p _n,3 represents the current (nth) phoneme, p _n,4 represents the phoneme one step back (subsequent phoneme), and p _n,5 represents the phoneme two steps back (subsequent phoneme). _{a n} represents information on accent type and position. b _n represents information on the part of speech, inflection form, and inflection type of the preceding word. c _n represents information on the part of speech, inflection form, and inflection type of the current word. d _n represents information on the part of speech, inflection form, and inflection type of the subsequent word. e _n represents information on the preceding accent phrase. f _n represents information on the current accent phrase. g _n represents information on the subsequent accent phrase. h _n represents information on the preceding breath paragraph. i _n represents information on the current breath paragraph. j _n indicates information on the following breath group, and k _n indicates the number of breath groups, accent phrases, and moras (sound segments) in the utterance.

In this way, the context information includes information on the phoneme in the utterance, information on the phonemes before and after the phoneme, accent phrase information on the phoneme, etc. The accent phrase information indicates features related to the accent phrase in the utterance in which the current phoneme is included, and features related to the accent phrase adjacent to the accent phrase, etc. The position is represented by taking the position of the current phoneme as "0", with positions before the current phoneme being represented by negative values and positions after the current phoneme being represented by positive values.
An example of the format of the context information is shown in Fig. 10. Context information _Ln in Fig. 10 indicates information on the n-th phoneme in the phoneme string.

Returning to FIG.
The context information generating means 34 generates context information L _n (n=1 to N, N: number of phonemes) shown in FIG. 10 for each phoneme from the section text data (a mixture of kanji and kana text).
A common method can be used to generate context information from text data of mixed kanji and kana sentences. For example, the technology described in the reference "Open JTalk, [online], [searched March 6, 2020], Internet <http://open-jtalk.sourceforge.net/>" can be used. This technology has a morphological analysis function, an accent dictionary function, and other language processing functions, and reflects information from the mixed kanji and kana sentences in the form of context labels to each label.
The context information generating means 34 outputs the generated context information of the phoneme string to the converting means 35 .

The conversion means 35 converts the context information of the phoneme string generated by the context information generation means 34 into text data (second section text data) including characters representing the reading of the order in which the phonemes appear and predetermined characters representing the rhythm.
Here, the conversion means 35 converts the context information into PLP data (reading kana and prosodic symbols).
The conversion means 35 sequentially extracts phonemes p _n,3 (n=1 to N, N: number of phonemes) (see FIG. 8) from context information L ₁ , ..., L _n , ..., L _N (N: number of phonemes) of the phoneme string to generate a phoneme string.
Then, the conversion means 35 inserts a predetermined prosodic symbol after p _n,3 when a predetermined condition is met.
Specifically, the conversion means 35 inserts a predetermined prosodic symbol when the conditions (1) to (6) shown in FIG. 11 are met (see FIG. 8 and FIG. 9 as appropriate).

Condition (1) is a condition where a _n,3 = 1 in the context information L _n and a _n+1,2 = 1 in the context information L _n+1 . _{a n,3} means the position from the end of the current mora in the current accent phrase. In other words, a _n,3 = 1 indicates that the current mora position is the endmost position in the current accent phrase. _{a n,2} means the position from the beginning of the current mora in the current accent phrase. In other words, a _n+1,2 = 1 indicates that the current mora position is the beginning of the current accent phrase when the position of the following phoneme is the current position.
If this condition (1) is satisfied, the conversion means 35 inserts a prosodic symbol ("#") indicating the delimitation of the accent phrase after the phoneme pn _,3 .

Condition (2) is a condition that a _n,1 = 0 and a _n,2 ≠ f _n,1 of the context information L _n . _{a n,1} = 0 indicates that the accent type (position of the accent kernel) and the current mora position in the current accent phrase match. _{a n,2} ≠ f _n,1 indicates that the number of moras in the current accent phrase and the position from the beginning of the current mora in the current accent phrase do not match. In other words, it indicates that the phoneme of the context information L _n is not the last mora in the current accent phrase.
If this condition (2) is met, the conversion means 35 inserts a prosodic symbol ("&") indicating a falling accent after the phoneme _pn,3 .

Condition (3) is a condition where a _n,2 = 1 in the context information L _n and a _n+1,2 = 2 in the context information L _n+1 . _{a n,2} indicates the position of the current mora from the beginning of the current accent phrase. _{a n,2} = 1 indicates that the current mora position is the beginning in the current accent phrase. Also, a _n+1,2 = 2 indicates that the current mora position is the second in the current accent phrase when the position of the following phoneme is the current position.
If this condition (3) is satisfied, the conversion means 35 inserts a prosodic symbol ("") indicating an accent rise after the phoneme _pn,3 .

Condition (4) is a condition that the phoneme p _n,3 of the context information L _n is “pau”, which indicates a pause.
If this condition (4) is met, the conversion means 35 deletes "pau" from the phoneme _pn,3 and inserts a prosodic symbol ("_") indicating a pause.

Condition (5) is a condition that the phoneme p _n,3 of the context information L _n is "sil" representing silence, n=N, and e _n,3 =0. n=N indicates that the current phoneme is the last phoneme in the utterance. _{e n,3} =0 indicates that the intonation at the end of the sentence is a normal intonation that is not a question.
If this condition (5) is satisfied, the conversion means 35 deletes the "sil" of the phoneme _pn,3 and inserts the prosodic symbol ("(") which indicates the end of the sentence (normal).

Condition (6) is that the phoneme _pn,3 of the context information _Ln is "sil" representing silence, n=N, and e _n,3 =1. n=N indicates that the current phoneme is the last phoneme in the utterance. _{e n,3} =1 indicates that the final intonation is an interrogative intonation.
If this condition (6) is met, the conversion means 35 deletes the "sil" of the phoneme _pn,3 and inserts a prosodic symbol ("?") indicating the end of the sentence (a question).

As a result, the conversion means 35 converts the context information _L1 , ..., _Ln , ..., _LN (N: number of phonemes) into PLP data (PLP _N ), which is text data in which prosodic symbols are inserted into a phoneme sequence _p1,3 , _p2,3 , ..., pN _,3, as shown in Figure 12.
In this example, the reading of the PLP data is expressed by phoneme symbols (phoneme symbols indicating p1 _{, 3} , etc.), but the conversion means 35 may convert the phoneme symbols into hiragana, katakana, phonetic symbols, roman letters, etc. When converted into katakana, the PLP data becomes as shown in FIG.

Returning to FIG.
The conversion means 35 transmits the section audio data separated by the matching means 33 and the section PLP data (second section text data) which is the PLP data corresponding to that section, to the editing terminal 4 via networks N and _N2 using a communication means not shown.

With the above-described configuration, the speech text generation device 3B can generate section speech data and section PLP data associated with each utterance as learning data from the speech data and text data. At this time, the speech text generation device 3B can associate the section speech data, which is the speech included in the speech data, with the section PLP data corresponding to the subtitle data regardless of the time lag.
The speech text generation device 3B can be operated by a learning data generation program for causing a computer (not shown) to function as each of the above-mentioned means.

In the learning data generation system 100B, the editing terminal 4 stores voice data (section voice data) and text data (PLP data) for each speech section in the learning data storage means 40. Then, as shown in Fig. 13, the editing terminal 4 displays the section PLP data on a text correction screen g11 of the editing screen G2 and performs correction.
The speech text generation device 3B may transmit the section text data to the editing terminal 4 together with the section voice data and the section PLP data.
In this case, the editing terminal 4 displays the editing screen G2B as shown in FIG. 14 by the modifying means 41, and the section text data and the section PLP data can be subject to modification.
In the example of FIG. 14, the text correction means 411 displays two text correction screens (g11a, g11b), and the section text data is corrected on the text correction screen g11a, and the section PLP data is corrected on the text correction screen g11b.

<Operation of the learning data generation system>
Next, the operation of the training data generation system 100B (a method for generating spoken voice text) according to an embodiment of the present invention will be described with reference to FIG. 15 (for the configuration, refer to FIG. 5 as appropriate).
The operations from steps S1 to S7 are the same as those of the learning data generation system 100 described with reference to FIG. 4, and therefore will not be described here.

In step S7A, the speech text generation device 3B uses the context information generation means 34 to perform morphological analysis and linguistic analysis on the section text data (mixed kanji and kana sentences) associated with the section speech data in step S7, thereby generating context information (context label data) for each phoneme from the section text data.
In step S7B, the speech text generation device 3B extracts a phoneme string from the context information generated in step 7A using the conversion means 35, and adds prosodic symbols in accordance with the conditions shown in FIG. 11 to convert the context information of the phoneme string corresponding to the section speech data into PLP data (section PLP data; second section text data).

In step 8A, the speech text generation device 3B transmits the generated training data (section speech data, section PLP data) to the editing terminal 4, and the editing terminal 4 stores the section speech data and section text data in association with each other in the training data storage means 40.
In step S9B, the editing terminal 4 uses the modifying means 41 to modify the delimiter positions of the section audio data and the character strings of the section PLP data, as necessary, according to the operator's decision.
Here, the editing terminal 4 uses the voice segment correction means 410 to correct segment positions of the section voice data, and uses the text correction means 411 to correct the section PLP data.
Through the above operations, the training data generation system 100B can generate training data for training a DNN model used for speech synthesis or speech recognition.

100, 100B Learning data generation system 1 Subtitled data storage device 2 Upload terminal 20 File selection means 21 File separation means (separation means)
22 Broadcast data receiving means 23 Broadcast data separating means (separating means)
3, 3B Speech text generating device 30 Speech text storage means 31 Speech segment detection means 32 Speech recognition means 33 Matching means 34 Context information generation means 35 Conversion means 4 Editing terminal 40 Learning data storage means 41 Correction means 410 Speech segment correction means 411 Text correction means

Claims (6)

A voice segment detection means for detecting segment positions of section voice data for each utterance from voice data consisting of a plurality of utterances;
a voice recognition means for performing voice recognition for each of the section voice data;
a matching means for estimating section text data corresponding to a time of the section voice data by matching a recognition result of the voice recognition means with text data which is a spoken content of the voice data;
a context information generating means for generating, from the section text data, context information for each phoneme, the context information including at least information on the phoneme and accent phrase information indicating characteristics related to an accent phrase including the phoneme and an accent phrase adjacent to the accent phrase;
a conversion means for converting the context information of the phoneme string into second section text data including characters representing the pronunciation of the phonemes in the order of appearance and predetermined characters representing prosody;
A speech text generation device comprising: 2. The speech text generation device according to claim 1, further comprising a separation means for separating the speech data and the subtitle data that becomes the text data from subtitled data including a plurality of speech sounds and subtitles corresponding to the speech sounds. 3. The speech text generation device according to claim 1, further comprising a speech segment correction means for correcting the segment positions of the section speech data based on an operation by an operator. 4. The speech text generating device according to claim 1, further comprising a text correction unit that corrects the section text data based on an operation by an operator. A speech text generation program for causing a computer to function as the speech text generation device according to any one of claims 1 to 4 . detecting a segmentation position of section voice data for each utterance from voice data consisting of a plurality of utterances by a voice segmentation detection means;
performing voice recognition for each of the section voice data by a voice recognition means;
a step of estimating section text data corresponding to a time of the section voice data by matching a recognition result of the voice recognition means with text data which is the spoken content of the voice data by a matching means;
generating, from the section text data, context information for each phoneme, the context information including at least information on the phoneme and accent phrase information indicating characteristics related to an accent phrase including the phoneme and an accent phrase adjacent to the accent phrase;
converting the context information of the phoneme string into second section text data including characters representing the pronunciation of the phonemes in the order of appearance and predetermined characters representing prosody by a conversion means;
A method for generating speech to text, comprising:

Images