JPH08212190A - Support device for production of multimedia data - Google Patents

Abstract

PURPOSE: To provide a support device for production of multimedia data which can automatically and effectively secure the correspondence between a text and the speech signals for every sentence that is punctuated at pauses or every phrase, for example. CONSTITUTION: The language processing is applied to the text data at a speech feature estimation part 1, and the part 1 estimates the series of a time when a silent section of a fixed length or longer alternates with other sections of the voices obtained when the text data are read aloud. Then a speech signal analysis part 2 processes the speech signals and extracts the series of a time when a silent section of a fixed length of longer alternates with other sections out of the speech signals. The correspondence is secured at a collation part 3 between both output series data, and the series collation result data are produced and outputted through an output part 4.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multimedia data creation support apparatus for supporting the creation of multimedia data in which voice information and text information are integrated, and more particularly to moving image data synchronized with voice or only voice. The present invention relates to a multimedia data creation support device that automatically synchronizes data and text corresponding to them.

[0002]

2. Description of the Related Art Conventionally, a process for automatically synchronizing voice moving image data and scenario text data has not been put into practical use. Therefore, this process had to be performed manually.

At the research level, there has been proposed a device for associating the audio / video data of a news program with the item table (news item table) of this program. With this technique,
Using two techniques of "image recognition" and "speaker identification", a series of audio moving image data is divided into the number of items existing in the item table, and each divided part and each item in the item table are divided. Is associated with.

There is also a technique called "automatic labeling of voice data" in which a voice signal and a phoneme sequence corresponding to the voice signal are given to automatically take correspondence between them.

[0005]

As described above, the manual operation of synchronizing the audio moving image and the scenario text is performed by comparing the moving image data and the scenario text one by one to which part of the former and which part of the latter. Since there is no choice but to visually identify whether or not this corresponds, a great deal of labor and time are required.

On the other hand, in the case of a device using "image recognition" and "speaker identification", the technology of "image recognition" and "speaker identification" on which it depends, especially the former technology, is still under development. Yes,
There are still unsolved problems in making it generic. Even if this problem is solved, since the minimum unit of correspondence is limited to the news item, it is not possible to make a finer range of correspondence.

Further, in the automatic labeling of speech data, since the unit of correspondence is basically phonological, only techniques for data of words or at most sentence length have been studied. For ten texts, it may not work or may not fit into practical calculation. Also, the correspondence of phoneme units is excessive for supporting the creation of multimedia data.

The present invention has been made in view of the above,
It is an object of the present invention to provide a multimedia data creation support device that automatically and efficiently associates text and audio signals with each other in units of sentences or phrases separated by pauses.

[0009]

In order to achieve the above object, a multimedia data creation support apparatus of the present invention is a multimedia data creation support apparatus that synchronizes texts with audio signals corresponding to the texts. hand,
Predicting means for predicting a sequence of times at which a silent section having a length of a certain length or longer and a section other than that in speech obtained when the text is read aloud by linguistically processing the text; By extracting a series of times at which a silent section of a certain length or more in the voice signal and other sections in the audio signal alternate, and output sequence data of the predicting section and the extracting section. The gist is to have a matching unit that associates with output series data and creates series matching result data.

Further, in the multimedia data creation support apparatus of the present invention, a silent section in the series predicted by the predicting section by the collating means and a section other than that in the series are extracted from the series by the extracting means. The gist is to have a means for associating the silent section and the other sections so that the evaluation value becomes maximum.

Further, in the multimedia data creation support apparatus of the present invention, as the evaluation value, the sections of the same type correspond to each other and are larger than the sections of different types correspond to each other. The gist is that the smaller the variation in the length ratio, the larger the value.

The multimedia data creation support apparatus of the present invention is characterized in that the voice signal is associated with the voice signal in the voice moving image signal, and the text is associated with the scenario text for the voice moving image.

[0013]

In the multimedia data creation support apparatus of the present invention, the predicting means performs language processing on the text, and the silent section of a certain length or longer in the voice obtained when the text is read aloud is replaced with the other section. Predicting the time series, processing the audio signal in the extraction means, extracting the time series at which the silent section of a certain length or more in the audio signal and the other sections alternate, and corresponding both output series data The collation means is used to create the series collation result data.

Further, in the multimedia data creation support apparatus of the present invention, the collation means has the maximum evaluation value for each section in the series predicted by the prediction means with respect to each section in the series extracted by the extraction means. Correspond to

Furthermore, in the multimedia data creation support apparatus of the present invention, the evaluation value is larger when the sections of the same type correspond to each other than when sections of different types correspond to each other. The smaller the variation in the length ratio, the larger the value.

In the multimedia data creation support apparatus of the present invention, the audio signal corresponds to the audio signal in the audio moving image signal, and the text corresponds to the scenario text for the audio moving image.

[0017]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of the multimedia data creation support apparatus of the present invention. In the figure, the speech feature prediction unit 1 constitutes the prediction means of the present invention, and comprises a pronunciation information creation unit 1a and a sequence data creation unit 1b, and by analyzing the input Kanji-Kana mixing text, A phonological sequence, a pause, and the like for uttering the text are determined, and using this, an alternating sequence of "silent intervals of a certain length or longer" and "other intervals" is created.

The pronunciation information creation unit 1a is a device that analyzes a kanji-kana mixing sentence and adds information necessary for pronunciation such as "reading", "pause" and "accent". This device is a device that is always included in a speech synthesis system using text as an input, so detailed description will be omitted. In this embodiment, only the "reading" and "pause" information of this output is transferred to the series data creation unit 1b.

The sequence data generation unit 1b outputs a sequence of times at which a voiced section and a silent section are switched based on the "reading" information and the "pause" information. Hereinafter, this series is referred to as series A. The processing result is Ab (i), Ae.
(I) (1 ≦ i ≦ n) is output as two array data. Here, n is the number of predicted voiced sections, and Ab (i) is the start time (Ab) of the i-th voiced section.
(1) = 0), Ae (i) holds the end time of the i-th voiced section. In this embodiment,
Although the number of seconds is used as the unit of time, another unit such as the number of mora may be used instead of this. Further, the series data creation unit 1b performs a process of extracting a portion of the input text corresponding to the i-th section and writing it in a predetermined storage area D (i).

The voice signal analysis section 2 constitutes the extraction means of the present invention, receives a digitized voice signal as an input, and uses a signal processing technique to output a silent section present in the voice signal. In descending order of m (longest time)
= N + l (l is an arbitrary number of 20% or more of n) are identified, and a sequence of time points at which the voiced section and the silent section are switched is output. Hereinafter, this series is referred to as series B.

Similar to the series A, the series B is Bb (i), Be
(I) (1 ≦ i ≦ m) is output as two array data. Here, m is the number of extracted voiced sections, and Bb (i) and Be (i) hold the start time and end time of the i-th voiced section, respectively. The start time of the first voiced section is 0, that is, Bb.
(1) = 0.

The process of creating such a sequence from a voice signal is a process of identifying a voiced section from voice signal data digitized by using a power value and the number of zero crossings (see, for example, Reference 1, Yasunaga Niimi, It can be easily implemented by using the Information Science Course "Voice Recognition", 1979.

The matching section 3 constitutes a matching means, and which of the voiced sections output from the voice signal analysis section 2 corresponds to each voiced section output from the voice feature prediction section 1. Is determined so that the following "evaluation value regarding goodness of correspondence" is maximized.

The "evaluation value for goodness of correspondence" means the following 2
It is a value obtained by combining two evaluation scales.

(1) It is better that sections of the same type, that is, sections having sound, and sections having no sound correspond to sections of different types.

(2) The ratio of the lengths of a certain section and the section corresponding to this section is constant regardless of the section, that is,
The smaller the variation, the better.

In this embodiment, the voiced section of sequence A and the sequence B are sequenced.
The correspondence of the voiced section of is expressed by a function q (i). That is, as shown in FIG. 11, the value of the function q (i) indicates the number of the head of the voiced section of the sequence B corresponding to the i-th voiced section of the sequence A. Further, the number of the end of the voiced section of the series B corresponding to the i-th voiced section of the series A is represented by q (i + 1) -1. For example, when q (i) = j, the start time Ab of the i-th voiced section of the sequence A
(I) and the start time Bb of the j-th voiced section of the sequence B
It corresponds to (j). Furthermore, when q (i + 1) = k, the end time Ae of the i-th voiced section of the sequence A
Be (k-1) corresponds to (i).

Note that q (i) must satisfy the following constraint (constraint for preventing the context of corresponding elements from being reversed).

Ifi <j then q (i) ≦ q (j) (1) The evaluation values in this example are as follows.

[0031]

[Equation 1] The above equations will be described below.

The first term in the equation (2) represents the evaluation value of the degree of correspondence regarding the section from the beginning of the series A to the start time of the nth voiced section. This evaluation value is calculated with one voiced section of the sequence A and a silent section immediately after that as a unit section. The i-th unit section is from Ab (i) to Ab (i + 1),
The section of the sequence B corresponding to this is from Bb (q (i)) to Bb.
(Q (i + 1)). The calculation formula of the evaluation value for one unit section is defined as the formula (3) using i, q (i) and q (i + 1) as arguments.

Equation (3) is obtained by calculating the interval from Ab (i) to Ab (i + 1) in the sequence A and the corresponding interval in the sequence B, that is, j1 = Bb (q (i)) to j2 = Bb.
(Q (i + 1))) is defined as an evaluation value for the interval up to (q (i + 1)), and the first term is a part of the sequence B corresponding to the i-th silent segment of the sequence A, that is, Be (j2-1). ) To B
This is an evaluation value related to the length of b (j2) (the longer the better as shown in equation (5)), the second term is the degree of correspondence of the i-th voiced section of the sequence A. Here, the "correspondence degree with respect to the i-th section of the series A" is a scale showing the goodness of correspondence between the length of the i-th section of the series A and the corresponding section B of the series A, and When the length of i is x and the length of the section of the sequence B corresponding to this is y, it is defined as in Expression (4). This measure calculates how far the ratio of the two intervals is from the ratio of the lengths of the entire series A and B, and adds a negative sign to it.

The second term of the equation (2) is an equation for calculating the evaluation value regarding the correspondence regarding the nth voiced section of the sequence A. Unlike the voiced sections up to that point, the n-th voiced section is not a pause between sentences in the immediately following silence section, so that only the voiced section is exceptionally evaluated as shown in the equation.

The output unit 4 combines the correspondence (q (i)) of the two sequences obtained by the collation unit 3 and the "correspondence data between the sequence and the text" transferred from the speech feature prediction unit 1. As a result, "correspondence between text and voice signal" is output.

A specific description will be given below with reference to an example.

Now, assume that the scenario text as shown in FIG. 2 is input. This text is phonetic information creation section 1a
FIG. 3 shows the result obtained by extracting only the information of “reading” and “pause” from the result of the processing performed by the above. This data is "notation" and "reading (phoneme series)"
It is a sequence of records consisting of and "English symbols L, M, S, N representing a pose". The pause column indicates the type of pose that should be inserted immediately after uttering "reading".
S and N correspond to a long pose, a middle pose, a short pose, and no pose, respectively, in this order.

Next, the processing of the series data creating section 1b will be described with reference to FIG.
This will be described with reference to the flowchart shown in FIG. First, step S40
At, the variables are initialized. i is a counter that holds the number of the voiced section of the input character string at the present time, and the initial value is 1. t is a variable that holds the time required to utter a given character string up to now, LP, MP, and SP are long pose, middle pose, and
ML, a constant that defines how many mora a short pose corresponds to
Is a constant that defines how many unit times one mora corresponds to. Note that the values of LP, MP, SP and ML in the figure are examples.

After the initialization is completed, it is checked in step S41 whether or not the next record is given from the pronunciation information creating section 1a. If not, it is determined that the text is finished and the process is finished. If the next record exists, it is read (step S42).

In step S43, it is determined whether or not a new voiced section is started by checking the value of the variable new, and if so, Ae (i) (the start time of the i-th voiced section is set). The value of t is substituted into the retained array element) (step S44). At this time, the value of the variable new is cleared at the same time.

In step S45, the number of moras is calculated by referring to the reading field of the input record. The number of mora is the number of characters in this field excluding "" (voiced sound symbol) and "°" (semi-voiced sound / nasal voiced sound symbol). Also, the characters existing in the notation field are added to the element of D (i).

In step S46, it is checked whether or not the input character is the letter L, and if so, the process proceeds to step S47. In step S47, the value of t is written in Ae (i), which represents the end time of the i-th voiced section, and then the increment of i and the time required for a long pause are added to t. Also, at this point, the i-th voiced section has ended, and a new voiced section starts from the next record, so the value of new is set to 1.

If the input character is M or S (step S48 or S50), the time of the middle pause or the short pause is added to t accordingly (step S49 or S51).

When this processing is executed by inputting a record string as shown in FIG. 3, arrays Ab (i), Ae (i), D
The value of (i) is as shown in FIG.

The voice signal analysis unit 2 of FIG. 1 identifies a voiced section by performing a simple signal analysis process using the voice signal as shown in FIG. 6 as an input, and Bb (j), Be as shown in FIG.
(J) (0 ≦ j ≦ m) is output.

FIG. 8 is a flow chart showing the processing process of the collating unit 3 in FIG. The method shown here is an application of the matching algorithm by the dynamic programming method, and q (i) (1 ≦ i ≦ n) representing the optimum association can be efficiently obtained.

The basic idea is that "i and j are taken out from the head of the series A and B respectively and the optimum correspondence is obtained, the correspondence is the part of the correspondence for optimally matching the entire series AB. Has become. " This way of thinking is correct when the calculation formula of the evaluation value regarding the association of the two series satisfies a certain condition, but the evaluation value of the present embodiment satisfies this condition.

First, in step S80, various variables are initialized. For p (i, j), the sections from the beginning to Ab (i) and Bb (j) of the sequence obtained from the speech feature prediction unit 1 and the sequence obtained from the speech signal analysis unit 2 corresponded to each other optimally. It is the evaluation value in the case.

First, in step S81, the voiced sequence A
From the beginning (Ab (1)) to Ab (2) and the sequence B
The evaluation value regarding the correspondence with the section from the beginning to Bb (j) is obtained. This calculation is performed for each j of 2 ≦ j ≦ m. Here, h (i, j, k) in the figure is the equation (3) already described in this embodiment.

In step S82, p (i, j) is obtained while changing i, j. From the principle of dynamic programming, p
(I, j) is the value of p (i-1, k) when k is moved within the range of (q (i-1) ≤ k ≤ j) and Ab (i-1) to Ab ( Section up to i) and Bb
It is the maximum value of the sum of the evaluation values between the section (k) and Bb (j), and the value of k at this time is q (i
-1).

Step S83 is an exceptional process for handling the final section of series 2. In the process of step S82, 1 is set from the start time of a certain voiced section to the start time of the next voiced section.
Although it is treated as one unit, since the last voiced section does not include a silent section after that, the calculation is performed so that the silent section is not included. F appearing in this step is the equation (4) already described in this embodiment.

The result of this process is shown in FIG.

The output unit 4 includes q (i) shown in FIG. 9 and FIG.
I, D (i), and Bb (i), Be (i) in FIG. 7, the start time and end time of the audio signal for each record in FIG. 5 are determined as follows.

For the i-th record other than the last record in FIG. 5, the start time is Bb (q (i)) and the end time is Be (q (i + 1) -1).

For the last record in FIG. 9, the start time is Bb (q (n)) and the end time is Bb (m).

The results are shown in FIG. FIG. 10 shows the start time and end time of the section of the voice signal corresponding to the text delimited by the silent section, and it can be seen that the voice signal and the text data are efficiently associated with each other.

In this embodiment, the case where the input data is the voice signal and the text data has been described, but the voice signal is the voice signal in the voice moving image signal and the text data is the scenario text data for the voice moving image. By substituting, a multimedia data creation support device for associating a voice moving image with a scenario text corresponding thereto can be realized in the same manner.

[0058]

As described above, according to the present invention,
Predict a sequence of times at which a silent section of a certain length or more and other sections in the speech obtained when reading the text are alternated, and predict a silent section of a certain length or more in the speech signal and other sections. A series of times at which the sections are changed is extracted, and both output series data are associated with each other by the collating means. Further, in the association by the collating means, the evaluation values are associated with each other so that the evaluation value becomes maximum, and the evaluation values are Since it is set that the corresponding sections of the same type are larger than the sections of different types correspond to each other, and the variation in the length ratio of the corresponding sections is smaller, the value is larger. Correspondence between a voice signal and text data can be automatically and efficiently performed without relying on human hands. Also, in terms of the amount of calculation, the number of silent sections having a length equal to or greater than a certain value is about several tens for one minute of voice, which is efficient.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a multimedia data creation support device according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example of input text.

FIG. 3 shows a result obtained by processing only the reading and pause information in addition to the original text from the result of processing the text shown in FIG. 2 in the pronunciation information creation unit used in the multimedia data creation support device of FIG. FIG.

FIG. 4 is a flowchart showing a process of a series data creation unit used in the multimedia data creation support device of FIG.

5 is a diagram showing an output of a speech feature prediction unit obtained as a result of the processing shown in FIG.

6 is a diagram showing an audio signal input to an audio signal analysis unit used in the apparatus of FIG.

FIG. 7 is a diagram showing an output from an audio signal analysis unit.

8 is a flowchart showing a process of a collating unit used in the multimedia data creation support device shown in FIG.

9 is a diagram showing an output from a collating unit used in the multimedia data creation support device shown in FIG.

10 is a diagram showing an output of the multimedia data creation support device shown in FIG. 1. FIG.

11 is an explanatory diagram showing a relationship between a start time and an end time in each of the series A and B. FIG.

[Explanation of symbols]

 1 Speech Feature Prediction Section 1a Pronunciation Information Creation Section 1b Series Data Creation Section 2 Speech Signal Analysis Section 3 Matching Section 4 Output Section

Claims (4)

[Claims] 1. A multimedia data creation support apparatus for synchronously associating a text with a voice signal corresponding to the text, the voice being obtained when the text is read aloud by linguistically processing the text. Prediction means for predicting a sequence of times at which a silent section of a certain length or more and another section alternate, and a silent section of a certain length or more in the audio signal by processing the audio signal, Extraction means for extracting a series of times at which other sections alternate with each other, matching means for associating the output series data of the prediction means with the output series data of the extraction means, and creating the series matching result data, An apparatus for supporting multimedia data creation, comprising: 2. The matching unit compares the silent section in the sequence predicted by the predicting unit and the other sections with respect to the silent section in the series extracted by the extracting unit and other sections. The multimedia data creation support apparatus according to claim 1, further comprising means for associating the evaluation value with a maximum value. 3. The evaluation value is larger as the sections of the same type correspond to each other than the sections of different types correspond to each other, and the variation in the ratio of the lengths of the corresponding sections is smaller. The multimedia data creation support apparatus according to claim 2, wherein 4. The audio signal in the audio moving image signal is associated with the audio signal, and the scenario text for the audio moving image is associated with the text. Multimedia data creation support device.