JP2003259311A - Video reproducing method, video reproducing apparatus, and video reproducing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video reproducing method for compressing the contents of a recorded program at an optional summary rate and reproducing the summary of the contents. <P>SOLUTION: The video reproducing method adopting an audio processing method for deciding an emphasis state on the basis of a feature quantity resulting from analyzing an audio signal by each frame, uses a code table for storing an incident probability in the emphasis state and an incident probability in a still state in cross-reference with each other to obtain the incident probability in the emphasis state corresponding to the feature quantity resulting from analyzing an audio signal by each frame and the incident probability in the still state, calculates the probability of incidence of the emphasis state on the basis of the incidence probability of the audio signal to be summarized recorded on a recording medium and the probability of incidence of the still state on the basis of the incidence probability in the still state, decides an audio signal period wherein a probability ratio of the probability of incidence in the emphasis state to the probability of incidence in the still state is greater than a prescribed coefficient to be a summarized period, calculates the total sum of times in the summarized period, decides the summarized period wherein the summarized time is nearly a prescribed summarized time, and reads the decided summarized period from the recording medium and reproduces it. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video reproducing method, a video reproducing apparatus and a video reproducing program for reproducing various video with audio recorded on a recording medium in a summarized manner.

[0002]

2. Description of the Related Art Conventionally, various summarizing methods have been proposed. As one of them, there is a device that extracts a section moving image consisting of a plurality of continuous frames from each block of the entire moving image and connects the extracted section moving images of each block to form a digest image. For example, Japanese Patent Laid-Open No. 8-93
No. 10, Japanese Unexamined Patent Publication No. 3-90968, Japanese Unexamined Patent Publication No. 6-165509, and the like.
Also, as a time compression method for audio segments, there has been a method in which the ratio of pause compression is precisely controlled to create a digest with high intelligibility. For example, Japanese Patent Laid-Open No. 20
No. 01-154700, etc.

There has also been a system in which a telop or sound information is used to extract a characteristic scene or scene of the program video to make it into a digest video. For example, Japanese Patent Laid-Open No. 2
No. 001-23062.

[0004]

In order to summarize the content at an arbitrary time or generate a digest, it is necessary to previously determine the priority order of each scene constituting the content. JP-A-8-9310,
JP-A-3-90968, JP-A-6-
In Japanese Patent No. 165009, a user thinks that a scene is important by using a pointing device such as a joystick or a plurality of buttons to add digest priority information. The burden on the user for generating the digest is large.

Further, Japanese Patent Laid-Open No. 2001-154700
In the official gazette, the digest is generated by the pause compression. However, since most of the content is normally occupied by the non-pause section, simply removing the pause makes the summary reproduction time 1/10 of the original content reproduction time. It is unrealistic to compress the content at a high compression rate as described above. Further, in Japanese Unexamined Patent Publication No. 2000-23062, as a digest video generation method, a summary section specified only by a sound volume value of sound information is not necessarily an important section. This is because the volume is not always loud when the point is emphasized. Further, when the telop information is used, it is impossible to generate the digest of the content in which the telop does not exist or the digest in the section in which the telop does not appear.

[0006] In addition, when receiving and reproducing a real-time audio signal with a video such as a live broadcast, if the program cannot be viewed due to leaving the seat, etc., the first half of the program is summarized while continuing recording. If you can see it, you are expected to be able to watch the following video after understanding the scenario in the first half of the program. An object of the present invention is to propose a video reproducing method, a video reproducing apparatus, and a video reproducing program capable of compressing and reproducing a video stored in a recording medium at an arbitrary time.

[0007]

According to the present invention, a real-time video signal and an audio signal are stored in association with a reproduction time, a summary start time is input, and a summary time or a summary section which is a total extension time of the summary section. Enter the summarization rate, which is the ratio of the total extension time to all summarization target sections, and the summarization time or summarization rate is determined to be the emphasized state for the voice signal in the summarization target section from the summarization start time to the summarization end time to the present. We propose a video reproduction method for determining the section as a summary section and reproducing the audio signal and the video signal in the summary section.

In the present invention, further, the reproduction end time of the audio signal and the video signal of the digest section is set as the end time of the new digest section, and the reproduction end time of the digest section is set as the new digest section reproduction start time. We propose a video playback method that repeats the determination of the summary section and the playback of the audio and video signals in the summary section. The present invention further proposes a video reproducing method in which the summarization rate r (r is a real number satisfying 0 <r <1) is adjusted to r / (1 + r), and the summarization section is determined based on the adjusted summarization rate. Further, in the present invention, at least the fundamental frequency or the pitch period, the power, the time change characteristic of the dynamic feature amount, or the feature amount including the difference between the frames, the appearance probability in the emphasized state, and the appearance probability in the quiet state are associated. Using the codebook stored as above, the appearance probability in an emphasized state corresponding to the feature amount obtained by analyzing the voice signal for each frame is obtained, and the voice signal is obtained at a quiet state corresponding to the feature amount analyzed for each frame. The probability of appearance is calculated, the probability of becoming an emphasized state is calculated based on the appearance probability of the emphasized state, the probability of becoming a calm state is calculated based on the appearance probability of the calm state, and the probability of becoming the emphasized state is calculated. The probability ratio to the probability of being in the quiet state is calculated for each voice signal section, the time of the voice signal section corresponding to the probability ratio is accumulated in descending order to calculate the summarization time, and the total sum of the summarization time is calculated. The speech signal interval summarization rate which is the ratio of target section serving as the input summarization rate proposes a video reproducing method of determining with the summary section.

Further, according to the present invention, at least the fundamental frequency or the pitch period, the power, and the time change characteristic of the dynamic feature quantity,
Or, using a codebook in which the feature amount including these inter-frame differences, the appearance probability in the emphasized state, and the appearance probability in the calm state are stored in correspondence with each other, the voice signal corresponds to the feature amount analyzed for each frame. The appearance probability in the emphasized state is obtained, the appearance probability in the quiet state corresponding to the feature amount obtained by analyzing the voice signal for each frame is obtained, and the probability of becoming the emphasized state is calculated based on the appearance probability in the emphasized state. Then, the probability of being in a calm state is calculated based on the appearance probability in the calm state, and a probability signal ratio of the probability of becoming the emphasized state to the probability of becoming the calm state is larger than a predetermined coefficient as a summary section. A tentative determination is made, or the ratio of the time of all the voice signal sections to the sum of the times of the summary sections is calculated as the sum of the times of the summary sections or the summarization rate, and the sum of the times of the summary sections is a substantially predetermined summary. During, or the summarization ratio proposes a video reproducing method of determining the predetermined factor is calculated and summarization segment to which a substantially predetermined summarization ratio.

Further, according to the present invention, it is determined for each frame whether the voice signal is a silent section or a voiced section, and a portion surrounded by a voiceless section of a predetermined number of frames or more, and a portion including the voiced section is referred to as a voice sub-paragraph. It is determined that the average power of voiced sections included in the audio sub-paragraph is less than a predetermined constant multiple of the average power in the audio sub-paragraph, the audio sub-paragraph group ending with the audio sub-paragraph is determined as the audio paragraph, the audio The signal section is defined for each audio paragraph, and a video reproducing method for accumulating and calculating the summary time for each audio paragraph is proposed.

Further, according to the present invention, a storage means for storing the real-time video signal and the audio signal in association with the reproduction time, a summary start time input means for inputting the summary start time, and a summary which is a total extension time of the summary section. Summarization condition input means for inputting summarization conditions defined by the summarization rate, which is the ratio of all summarization target sections of time or total extension time of summarization sections, and according to the summarization conditions, from the summarization start time to the summarization end time to the present A summarization section determining means for determining a section determined to be an emphasized state with respect to the audio signal in the summarization target section, and a reproducing means for reproducing the audio signal and the video signal of the summarization section determined by the summarization section determining unit. A video reproducing apparatus having the above is proposed.

According to the present invention, further, in the video reproducing apparatus, the summarizing section determining means has at least a fundamental frequency or pitch period, power, and a time change characteristic of a dynamic feature quantity,
Alternatively, a codebook that stores the feature quantity including these interframe differences, the appearance probability in the emphasized state, and the appearance probability in the quiet state in association with each other, and the speech signal is analyzed for each frame using the codebook. Obtaining the appearance probability in the emphasized state corresponding to the feature amount, an emphasized state probability calculation unit that calculates the probability of becoming the emphasized state based on the appearance probability in the emphasized state, and the voice signal framed using the codebook. Obtaining the appearance probability in a quiet state corresponding to each feature amount analyzed for each, a quiet state probability calculation unit that calculates the probability of being in a quiet state based on the appearance probability in the quiet state, and the probability of becoming the emphasized state The probability ratio to the probability of being in the quiet state is calculated for each voice signal section, the time of the voice signal section corresponding to the probability ratio is accumulated in descending order to calculate the summary time, and the summary section for temporarily determining the summary section Temporary And tough, the ratio to the total summary target section of the summary section proposes a video reproducing apparatus and a summary section determining unit that determines said summarization segment the speech signal segment that satisfies said summarization rate.

According to the present invention, further, in the video reproducing apparatus, the summarizing section determining means has at least a fundamental frequency or pitch period, power, and a time change characteristic of a dynamic feature quantity,
Alternatively, a codebook that stores the feature quantity including the inter-frame difference, the appearance probability in the emphasized state, and the appearance probability in the quiet state in association with each other, and the speech code is analyzed for each frame using the codebook. An appearance probability in an emphasized state corresponding to a feature amount and an appearance probability in a calm state are obtained, and an emphasized state probability calculation unit that calculates a probability of becoming an emphasized state based on the appearance probability in the emphasized state, and in the calm state A calm state probability calculation unit that calculates the probability of being in a calm state based on the occurrence probability of
A summary section tentative determination unit that tentatively determines that a speech signal section whose probability ratio of the probability of being in the emphasized state to the probability of being in the quiet state is larger than a predetermined coefficient is a summary section, and a summation of the time of the summary sections is approximately a predetermined summary. A video reproduction device having a digest section determining unit that determines the digest section for each channel or each speaker by calculating the predetermined coefficient at which the summarization rate becomes a substantially predetermined summarization rate.

The present invention further proposes a video reproducing program which is described by a computer-readable code and executes any one of the above-mentioned video reproducing methods. Effect According to the video reproducing method of the present invention, the voice section recorded in the recording medium, which has a high probability of being in the emphasized state, is extracted as the summary section. Therefore, an important part is extracted from the content and the important part is extracted. It is possible to obtain the summary audio and the summary video information by connecting them. As a result, even if the summarization time is compressed in a short time, the content can be well understood.

Further, since a recording medium capable of writing and reading a large amount of data in a short time is used as the recording medium, it is possible to read other contents or the first half of the contents being recorded while continuing recording.
Therefore, while continuing the recording, the recorded portion of the program being recorded can be summarized and the summarized information can be reproduced. As a result, it is possible to view both the current broadcast content and the past broadcast content by displaying the video being recorded and the video that conveys the summary on different display means from the main screen and the sub screen, respectively. When the reproduction of the summary information is finished, there is an advantage that the continuation can be viewed while understanding the first half of the program.

A feature of the present invention is to use a summarization method capable of summarizing contents at any summarization rate (compression rate) according to a request from a user at the time of content summary reproduction. This characteristic summarizing method determines the utterance state of an arbitrary speech sub-paragraph proposed by the present applicant in Japanese Patent Application No. 2001-241278, which is a prior application, and the probability of being in an emphasized state is higher than the probability of being in a quiet state. If it is large, it can be realized by using the voice emphasis state determination method and the voice summarization method, which determine that the voice sub-paragraph is in the emphasized state and extract the voice paragraph including the voice sub-paragraph as a summary section.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION Here, a method for extracting a voice sub-paragraph, a method for extracting a voice paragraph, and a method for obtaining a probability of being in an emphasized state and a probability of being in a quiet state for each voice sub-paragraph used in the present invention will be described. . FIG. 5 shows the basic procedure of an embodiment of the previously proposed voice summarization method. In step S1, the input voice signal is analyzed to obtain a voice feature amount. Step S2
Then, an audio sub-paragraph of the input audio signal and an audio paragraph composed of a plurality of audio sub-paragraphs are extracted. In step S3, it is determined whether the frame forming each audio sub-paragraph is in a calm state, emphasized state, or uttered state. Based on this determination, a summary voice is created in step S4 to obtain the summary voice.

An example of applying natural spoken language or conversational voice to the summary will be described below. As the voice feature amount, one that is more stable than the spectral information even in a noisy environment and is less likely to depend on the speaker is used. The fundamental frequency (f0), the power (p), the time variation characteristic (d) of the dynamic feature amount of the voice, and the pause time length (silent section) (ps) are extracted from the input voice signal as the voice feature amount. The method of extracting these speech feature amounts is, for example, “acoustic / acoustic engineering” (Sadahiro Furui, Modern Science Co., 1998), “speech coding” (Takehiro Moriya, Institute of Electronics, Information and Communication Engineers, 1998), “Digital”. Speech processing "(Sadahiro Furui, Tokai University Press, 1985),
"Sound analysis algorithm based on complex sine wave model" (Shigeki Sagayama, Ph.D. thesis, 1998). The change over time in the dynamic feature amount of voice is a parameter that is a measure of the speech rate, and is disclosed in Japanese Patent No. 2976998.
You may use the thing of the No. That is, the time variation characteristic of the LPC spectrum coefficient that reflects the spectrum envelope as the dynamic variation is obtained, and the speech rate coefficient is obtained based on the time variation. More specifically, the LPC spectrum coefficient C1 (t), ... d (t) = Σi = 1k [Σf = t-f0t + f0 [f
× Ci (t)] / (Σf = t-f0t + f0f2) 2 Here, f0 is the number of speech section frames before and after (not necessarily an integral number of frames but may be a fixed time section), and k is the LPC spectrum. The order is i = 1, 2, ... K. As the coefficient of the speech rate, the number of maximum points of the change in the dynamic feature amount per unit time or the rate of change per unit time is used.

In the embodiment, for example, 100 ms is set as one frame and the shift is set as 50 ms. An average fundamental frequency is calculated for each frame (f0 '). Regarding the power, similarly, the average power (p ') for each frame is obtained. Further, the difference between f0 ′ of the current frame and f0 ′ before and after ± i frames is taken as ± Δf0′i (Δ component). Similarly for power, p ′ of the current frame and p ′ before and after ± i frames
And the difference ± Δp′i (Δ component) is calculated. f0 ', ± Δ
Normalize f0'i, p ', ± Δp'i. In this standard, for example, f0 ′ and ± Δf0′i are divided by the average fundamental frequency of the entire voice waveform and standardized. These standardized values are represented as f0 ″ and ± f0 ″ i. Similarly, p ′ and ± Δp′i are also normalized by dividing by the average power of the entire speech waveform that is the target of speech state determination. In normalizing, it may be divided by an average power for each audio sub-paragraph and audio paragraph described later. These normalized values are p ″, ± Δp ″
Denote by i. The value of i is, for example, i = 4. The number of peaks of the dynamic measure, that is, the number of maximum points of the change in the dynamic feature amount, within ± T1 ms before and after the current frame, is counted (d
p). The Δ component (−Δd) between this and the dp of the frame including the time T2ms before the start time of the current frame in the section.
p) is calculated. The Δ component (+ Δdp) between the dp number of ± T1 ms and the dp of the frame including the time T3 ms after the end time of the current frame in the section is obtained. These, T
The values of 1, T2 and T3 are, for example, T1 = T2 = T3 = 450.
ms. The time length of the silent section before and after the frame is ± p
Let s. In step S1, each value of these audio characteristic parameters is extracted for each frame.

FIG. 6 shows an example of the voice sub-paragraph of the input voice and the method of extracting the voice paragraph in step S2. Here, the audio sub-paragraph is used as a unit for determining the utterance state. In step S201, a silent section and a voiced section of the input voice signal are extracted. The silent section is determined to be a silent section if the power of each frame is equal to or less than a predetermined power value, and the voiced section is
For example, if the correlation function for each frame is greater than or equal to a predetermined correlation function value, it is determined as a voiced section. The voiced / unvoiced decision is often made with the peak value of the autocorrelation function or modified correlation function by equating it with the characteristic of periodicity / non-periodicity. The autocorrelation function of the prediction residual obtained by removing the spectrum envelope from the short-time spectrum of the input signal is a modified correlation function, and voiced / unvoiced determination is performed depending on whether the peak of the modified correlation function is larger than a predetermined threshold value. The pitch period 1 / f0 (fundamental frequency f0) is extracted by the delay time giving the peak. For details of the extraction method of these sections, for example,
"Digital audio processing" (Sadahiro Furui, Tokai University Press,
1985) and the like. Here, it has been described that each voice feature amount is analyzed for each frame from the voice signal. Good.

In step S202, when the time of each silent section surrounding the voiced section is t seconds or more, a portion including the voiced section surrounded by the silent section is set as a voice sub-paragraph.
This t is, for example, t = 400 ms. Step S20
In 3, the average power of the voiced section, preferably in the latter half of this audio sub-paragraph, is compared with the constant β times the average power value BA of the audio sub-paragraph. If the former is smaller, then The audio sub-paragraph is determined as the final audio sub-paragraph, and the audio sub-paragraph after the immediately preceding final audio sub-paragraph to the currently detected final audio sub-paragraph is determined as the audio paragraph. FIG. 7 schematically shows a voiced section, a voice sub-paragraph, and a voice paragraph. The voice sub-paragraph is extracted under the condition that the time of the silent section surrounding the voiced section is t seconds. FIG. 7 shows audio subparagraphs j-1, j, and j + 1. Here, the speech subsection j is composed of n voiced sections, and the average power is Pj. As a typical example of the voiced section, the average power of the voiced section v included in the speech subsection j is pv. The voice paragraph k is extracted from the power of the voiced section in the latter half of the voice sub-paragraph j and the voice sub-paragraph. When the average of the average power pi of voiced sections from i = n-α to n is smaller than the average power Pj of the voice sub-paragraph j, that is, when Σpi / (α + 1) <βPj Expression (1) is satisfied, the voice sub-paragraph It is assumed that j is the last audio sub-paragraph of the audio paragraph k. However, Σ is from i = n−α to n. Α and β in the equation (1) are constants, and these are manipulated to extract a voice paragraph. In the example, α was 3 and β was 0.8. In this way, a group of audio sub-paragraphs between the adjacent final audio sub-paragraphs can be determined as an audio paragraph with the final audio sub-paragraph as a delimiter.

FIG. 8 shows an example of the speech subparagraph utterance state determination method in step S3 in FIG. Step S301
Then, the voice feature quantity of the input voice sub-paragraph is vector-quantized. For this purpose, a codebook in which at least two quantized speech feature quantities (codes) are stored is created in advance. Here, the voice feature stored in the codebook is compared with the voice feature of the input voice or the voice already obtained by analysis, and the distortion (distance) between the voice features in the codebook is minimized. It is conventional to specify the quantized speech feature amount to be used. FIG. 9 shows an example of how to create this codebook. The test subject listens to a large number of learning voices and labels the one in the quiescent state and the one in the emphasized state (S501).

For example, the reason why the test subject puts emphasis in the utterance is as follows: (a) utter a loud voice and extend the noun or conjunction. (B) Extend the beginning of the talk to insist on a topic change and collect opinions. (C) When raising a voice to emphasize an important noun, etc. (d) It is a high-pitched sound, but the voice is not so loud (e) While laughing bitterly, it falsifies the true intention Time (f) When the end of the word becomes high-pitched, like asking or asking for consent from the surroundings (g) Slowly and powerfully, when the end of the word becomes louder, just like pressing (h) The voice is loud and high, and interrupts Insist that you speak louder than the other party, (i) when you speak a real intention or secret that is loud in a loud voice, or when you say something important to a loud person normally (for example, voice Is small He said that he was sick and sick. In this example, the calm state is
It was assumed that the subject felt that the utterance was calm, not any of the above (a) to (i).

In the above description, the object to be determined as the emphasized state is the utterance, but the emphasized state can be specified by music. Here, in music with audio,
When trying to specify the emphasis state from the voice, the reasons for feeling emphasis are as follows: (a) loud voice and high voice (b) strong voice (c) high voice and strong accent (d) voice High voice, voice quality changes (e) voice is extended, and voice is loud (f) voice is high, voice is high and accent is strong (g) voice is voice, voice is high, and yell A high (h) voice with a high accent and a changing voice (i) a long voice, and a large voice with a high ending (j) a high voice and a long voice (k) a long voice, Also, screaming and high voice (l) Word rising, powerful (m) Slowly strengthening (n) Irregular tone (o) Irregular tone and high voice The emphasis state can be specified. The reasons for feeling the emphasis are as follows: (a) increase in power of the entire emphasized part (b) large difference in pitch between sounds (c) increase in power (d) change in number of musical instruments (e) change in tone and tempo Etc.

By creating a codebook based on these, not only utterance but also music can be summarized. For each label section in the quiet state and the emphasized state, the voice feature amount is extracted (S502) and parameters are selected (S50), as in step S1 in FIG.
3). A codebook is created by the LBG algorithm using the parameters in the label sections in the quiet state and the emphasized state (S504). For the LBG algorithm,
For example, (Y. Linde, A. Buzo and
R. M. Gray, “An algorithmic fo
r vector quantizer design
n, "IEEE Trans. Commun., vo
l. Com-28, pp. 84-95, 1980). The codebook size can be changed to 2 to the n-th power. For this codebook creation, it is preferable to use a voice feature amount standardized in voice sub-paragraphs or for each appropriate section longer than this, or the voice feature amount of the entire learning voice.

In step S301 in FIG. 8, the codebook is used to standardize the voice feature amount of the input voice sub-paragraph for each voice feature amount, and the standardized voice feature amount is collated for each frame or Vector quantization is performed to obtain a code (quantized speech feature amount) for each frame. The voice feature quantity extracted from the input voice signal at this time is the same as the parameter used for the codebook creation. In order to identify the speech sub-paragraph that includes the emphasis state, the likelihood of the utterance state is calculated using the code in the speech sub-paragraph.
Ask for calmness and emphasis. Therefore, the appearance probabilities of an arbitrary code (quantized speech feature amount) are obtained in advance for the case of a quiet state and the case of an emphasized state, and the appearance probability and the code are stored in a codebook as a set. In the following, an example of how to obtain this appearance probability will be described. Codes (obtained for each frame) of the speech feature amount of one section (label section) given a label in the learning speech used for creating the codebook described above are Ci, Cj, Ck, ... Cn in time series. , The probability that the label section α is in the emphasized state is Pα (e), and the probability that the label section is in the stationary state is Pα (n). Pα (e) = Pemp (Ci) Pemp (Cj | Ci)
... Pemp (Cn | Ci ... Cn-1) = Pemp (C
i) ΠPemp (Cx | Ci ... Cx-1) Pα (n) = Pnrm (Ci) Pnrm (Cj | Ci)
... Pnrm (Cn | Ci ... Cn-1) = Pemp (C
i) ΠPnrm (Cx | Ci ... Cx-1). However, Pemp (Cx | Ci ... Cx-1) is a conditional probability that Cx is in an emphasized state next to the code sequence Ci ... Cx-1, and Pnrm (Cx | Ci ... Cx-1) is similarly Ci ... Cx-. It is the probability that Cx will be in a calm state with respect to 1. However, Π is a product of x = i + 1 to n. In addition, Pemp (Ci) quantizes the learning speech in frames, counts the number of Ci in these codes existing in the portion labeled as the emphasized state, and counts the count value for the total number of codes (the number of frames) of all the learning speeches. Pnrm (Ci) is a value obtained by dividing the number of Cis present in the portion labeled as a quiescent state by the total number of codes.

In order to simplify each state probability of this label section α, in this example, using the N-gram model (N <n), Pα (e) = Pemp (Cn | Cn-N + 1 ... Cn-1 ) Pα (n) = Pnrm (Cn | Cn−N + 1 ... Cn−1). That is, N-1 past code strings Cn from Cn
The probability that Cn will be obtained as an emphasized state next to -N + 1 ... Cn-1 is Pα (e), and similarly, the probability value of N-gram is linear with the probability value of lower-order M-gram (N ≧ M). It is preferable to apply a linear interpolation method that interpolates to For example, N-1 past code strings Cn-N + 1 ... Cn from Cn
−1, the probability that Cn is obtained in a calm state is Pα
(N). All of the conditional probabilities of Pα (e) and Pα (n) are obtained from the quantized code strings of the labeled learning speech, which correspond to the quantized code strings of the speech feature amount of the input speech signal. May not be obtained from the learning voice. Therefore, a high-order (that is, a long code string) conditional probability is obtained by interpolating a single occurrence probability and a lower-order conditional occurrence probability. For example, N = 3 trigr
am, bigram of N = 2, unigram of N = 1
Is used to perform the linear interpolation method. Regarding N-gram, linear interpolation method, and trigram, for example, “Spoken language processing” (Kenji Kita, Satoshi Nakamura, Masaaki Nagata, Morikita Publishing, 19
96, 29). That is, N = 3 (trigram): Pemp (Cn | Cn-2
Cn-1), Pnrm (Cn | Cn-2Cn-1) N = 2 (bigram): Pemp (Cn | Cn-
1), Pnrm (Cn | Cn-1) N = 1 (unigram): Pemp (Cn), Pnr
m (Cn), the probability of occurrence of Cn in these three emphasized states, and the probability of occurrence of Cn in three calm states are respectively calculated by the following equations, Pemp (Cn | Cn-2Cn-1), Pm
We will calculate nrm (Cn | Cn-2Cn-1). Pemp (Cn | Cn-2Cn-1) = [lambda] emp1Pemp (Cn | Cn-2Cn-1) + [lambda] emp2Pemp (Cn | Cn-1) + [lambda] emp3Pemp (Cn) Formula (2) Pnrm (Cn | Cn-2Cn-1) = [lambda] nrmlPnrm (). Cn | Cn-2Cn-1) + [lambda] nrm2Pnrm (Cn | Cn-1) + [lambda] nrm3Pnrm (Cn) Formula (3) When the learning data of Trigram is N, that is, the codes are C1, C2 ,. ． ． When the CN is obtained, the re-estimation formulas for λemp1, λemp2, and λemp3 are as follows from the above-mentioned reference “Spoken Language Processing”. λemp1 = 1 / NΣ (λemp1Pemp (Cn | C
n-2C-1) / (λemp1Pemp (Cn | Cn-
2C-1) + λemp2Pemp (Cn | C-1) + λ
emp3Pemp (Cn))) λemp2 = 1 / NΣ (λemp2Pemp (Cn | C
-1) / (λemp1Pemp (Cn | Cn-2C-
1) + λemp2Pemp (Cn | C-1) + λemp
3Pemp (Cn))) λemp3 = 1 / NΣ (λemp3Pemp (Cn) /
(Λemp1Pemp (Cn | Cn-2C-1) + λe
mp2Pemp (Cn | C-1) + λemp3Pemp
(Cn))) where Σ is the sum of n = 1 to N. Similarly, λnrm1, λnrm2, and λnrm3 are obtained in the same manner.

In this example, the label section α is the number of frames N
When the code obtained in α is Ci1, Ci2, ..., CiNα, the probability Pα that this label section α is in the emphasized state
(E), the probability Pα (n) of being in a calm state is as follows: Pα (e) = Pemp (Ci3 | Ci1Ci2) ... Pemp (CiNα | Ci (Nα-1) Ci (Nα-2)) Formula (4) Pα ( n) = Pnrm (Ci3 | Ci1Ci2) ... Pnrm (CiNα | Ci (Nα-1) Ci (Nα-2)) Formula (5) is obtained. To enable this calculation,
m, unigram, and bigram are obtained for arbitrary codes and stored in the codebook. That is, the codebook stores a set of the voice feature amount of each code, the appearance probability in the emphasized state, and the appearance probability in the quiet state in this example. The appearance probability in the emphasized state is the voice feature amount. Only the probability of appearing in an emphasized state irrespective of the voice feature amount in the past frame (unigram: described as a single appearance probability) or this and the voice feature amount in the past frame to the voice feature amount in the current frame For each frame-based audio feature quantity sequence, the audio feature quantity is either a combination with the conditional probability of appearing in the emphasized state, and the appearance probability in the quiet state is also the same as that of the previous frame Probability of appearing in a quiet state irrespective of the voice feature amount (unigr
am: written as a single appearance probability) or this and the voice feature quantity appears in a quiet state for each voice feature quantity sequence in frame units from the voice feature quantity in the past frame to the voice feature quantity in the current frame. It is either a conditional probability to perform or a combination.

For example, as shown in FIG. 10, for each code C1, C2, ... Each state is stored as a set. In step S302 in FIG. 8, the likelihood of the utterance state is calculated for the calm state and the emphasized state from the probabilities stored in the codebook for the codes of all the frames of the input speech sub-paragraph. FIG. 11 shows a schematic diagram of the embodiment. Among the audio sub-paragraphs starting from time t, up to the fourth frame are indicated by ˜. As described above, here, the frame length is 100 ms and the frame shift is 50 ms, which is longer. Code Ci at frame number f and times t to t + 100, code C at frame number f + 1 and times t + 50 to t + 150
j is the frame number f + 2, time t + 100 to t + 2
At 00, code Ck is frame number f + 3, time t +
When the code Cl is obtained at 150 to t + 250, that is, when the code is Ci, Cj, Ck, Cl in the frame order,
The trigram can be calculated for frames with frame numbers f + 2 and above. Ps is the probability that the voice sub-paragraph s is in the emphasized state.
(E), the probability of being in a calm state is Ps (n)
The probabilities up to the frame are: Ps (e) = Pemp (Ck | CiCj) Pemp (Cl | CjCk) Formula (6) Ps (n) = Pnrm (Ck | CiCj) Pnrm (Cl | CjCk) Formula (7) Become. However, in this example, Ck,
Obtaining the individual appearance probabilities of the Cl emphasized state and the calm state,
The conditional probability that Ck appears next to Cj in the emphasized state and the stationary state, Ck is Ci, and Cj is C next to Cj.
The conditional probabilities that appear in the emphasized state and the calm state next to j and Ck, respectively, are obtained from the codebook as follows. Pemp (Ck | CiCj) = λemp1Pemp (Ck | CiCj) + λem p2Pemp (Ck | Cj) + λemp3Pemp (Ck) Formula (8) Pemp (Cl | CjCmpλ = λemp1Pemp (Cl | CjCk) + λemp (Pe) (λ | emp)) Cl) Formula (9) Pnrm (Ck | CiCj) = λnrm1Pnrm (Ck | CiCj) + λnr m2Pnrm (Ck | Cj) + λnrm3Pnrm (Ck) Formula (10) Pnrm (ClrCrCr) Prrm (Cr) Crn (CrCm) CnCrm) Cl | Ck) + λnrm3Pnrm (Cl) Expression (11) Probability Ps of being in the emphasized state up to the fourth frame shown in Expressions (6) and (7) using Expressions (8) to (11) above.
From (e), the probability Ps (n) of being in a calm state is obtained. Here, Pemp (Ck | CiCj) and Pnrm (Ck |
CiCj) can be calculated at frame number f + 2.

In this example, the audio sub-paragraph s has N frames.
When the code obtained in s is Ci1, Ci2, ..., CiNs, the probability Ps (e) that this speech subsection s is in the emphasized state
And the probability Ps (n) of being in a calm state is calculated by the following equation. Ps (e) = Pemp (Ci3 | Ci1Ci2) ... Pe
mp (CiNs | Ci (Ns-1) Ci (Ns-2)) Ps (n) = Pnrm (Ci3 | Ci1Ci2) ... Pn
rm (CiNs | Ci (Ns-1) Ci (Ns-2)) In this example, if these probabilities are Ps (e)> Ps (n), the audio subsection S is in the emphasized state, Ps (n). )> P
If s (e), it is in a calm state.

FIG. 12 shows a voice emphasis state determination device and voice using the voice sub-paragraph extraction method, the voice paragraph extraction method, and the method for obtaining the probability of the emphasis state and the probability of the calm state for each voice sub-paragraph described above. 1 illustrates an embodiment of a summarizing device. An input voice (input voice signal) whose voice emphasis state is to be determined or whose voice summary section is to be determined is input to the input unit 11. The input unit 11 also includes a function of converting an input audio signal into a digital signal as needed. The digitized audio signal is stored in the storage unit 12 as needed. The voice feature amount extraction unit 13 extracts the voice feature amount described above for each frame. The extracted voice feature amount is normalized by the average value of the voice feature amount as necessary, and the quantization unit 1
In 4, the voice feature amount of each frame is quantized with reference to the codebook 15, and the quantized voice feature amount is sent to the emphasis probability calculation unit 16 and the quietness probability calculation unit 17. The codebook 15 is, for example, as shown in FIG.

The appearance probability in the emphasized state of the quantized speech feature quantity by the emphasis probability calculation unit 16 is calculated by using the corresponding probability stored in the codebook 15, for example, equation (8) or (9). Calculated by Similarly, the quietness probability calculation unit 17 uses the corresponding probability that the appearance probability of the quantized speech feature amount in a quiet state is stored in the codebook 15, for example, according to Expression (10) or (11). Calculated. The appearance rate in the emphasized state, the appearance probability in the quiet state, and the voice feature amount of each frame calculated for each frame by the emphasis probability calculation unit 16 and the quietness probability calculation unit 17 are stored in the storage unit together with the frame number assigned to each frame. Store in 12.

The control of each of these units is sequentially performed under the control of the control unit 19. An embodiment of the voice summarization device is shown in FIG.
A broken line block is added to the solid line block.
That is, the audio feature amount of each frame stored in the storage unit 12 is sent to the silent section determination unit 21 and the sound section determination unit 22, and the silent section determination unit 21 determines whether each frame is a silent section or not. Also, the voiced section determination unit 22 determines for each frame whether or not it is a voiced section. The silent segment determination result and the voiced segment determination result are input to the audio sub-paragraph determining unit 23. Based on the silent section determination and the voiced section determination, the speech subsection determining unit 23 determines that a portion including a voiced section surrounded by a continuous silent section of a predetermined number of frames is a speech subsection based on the determination of the voiced section. To determine. The determination result of the audio sub-paragraph determining unit 23 is written in the storage unit 12, added to the audio data sequence stored in the storage unit 12, and the audio sub-paragraph number sequence is given to the frame group surrounded by the silent section. At the same time, the determination result of the audio sub-paragraph determination unit 23 is input to the final audio sub-paragraph determination unit 24.

In the final voice sub-paragraph determination section 24, the final voice sub-paragraph is detected by the method described with reference to FIG. 7, the final voice sub-paragraph determination result is input to the voice paragraph determination section 25, and the voice paragraph determination is performed. The section 25 determines that a part including a plurality of audio sub-paragraphs between the two final audio sub-paragraphs is an audio paragraph. This audio paragraph determination result is also written in the storage unit 12, and an audio paragraph sequence number is given to the audio small paragraph number sequence stored in the storage unit 12. When operating as a voice summarizing device, the emphasis probability calculation unit 16 and the quietness probability calculation unit 17 read the emphasis probability and the quietness probability of each frame forming each audio subparagraph from the storage unit 12, and the probability of each audio subparagraph is calculated. For example, it is calculated by equation (8) and equation (10). The emphasis state determination unit 18 compares the probability calculation values for each audio sub-paragraph to determine whether or not the audio sub-paragraph is in the emphasized state, and the summary segment extraction unit 26 determines even one audio sub-paragraph in the audio paragraphs. If there is one that is determined to be in the emphasized state, the audio paragraph including the audio sub-paragraph is extracted. The control of each unit is performed by the control unit 19.

From the above, it can be understood that the method of separating the speech waveform composed of speech into the speech sub-paragraphs and the speech sub-paragraphs, and the probability of being in the emphasized state and the probability of being in the quiet state can be calculated for each of the speech sub-paragraphs. Next, an embodiment relating to a voice processing method and a voice processing apparatus in which the summarization rate can be freely set and changed using each of the above-described methods will be described. FIG. 13 shows a basic procedure of the embodiment of the voice processing method. In this embodiment, the voice emphasis probability calculation process is executed in step S11 to obtain the emphasis probability and the quietness probability of the voice sub-paragraph.

In step S12, a summary condition input step S12 is executed. In this summarization condition input step S12, for example, information for prompting the user to input the summarization time or summarization rate or compression rate is provided, and the summarization time or summarization rate or summarization rate or compression rate is input. It is also possible to adopt an input method of selecting one from a plurality of preset summarization times or summarization rates and compression rates. In step S13, the operation of repeating the change of the extraction condition is executed, and the extraction condition satisfying the summarization time or the summarization ratio and the compression ratio input in the summarization condition input step S12 of step S12 is determined.

In step S14, the abstract extraction step is executed. In this abstract extraction step S14, the voice paragraph to be adopted is determined using the extraction condition determined in the extraction condition change step S13, and the total extension time of this voice paragraph to be adopted is calculated. In step 15, a summary reproduction process is executed,
The voice paragraph string extracted in the abstract extraction step S14 is reproduced. FIG. 14 shows details of the voice emphasis probability calculation step shown in FIG. In step S101, the speech waveform string to be summarized is separated into speech sub-paragraphs. In step S102, a voice paragraph is extracted from the voice sub-paragraph string separated in step S101. As described with reference to FIG. 7, the audio paragraph is a unit which is composed of one or more audio sub-paragraphs and whose meaning can be understood.

For each audio sub-paragraph extracted in step S101 in steps S103 and S104, each audio sub-paragraph is highlighted using the codebook described in FIG. 10 and the above equations (8) and (10). A probability (hereinafter referred to as an emphasis probability) Ps (e) and a probability of being in a calm state (hereinafter referred to as a calm probability) Ps (n) are obtained. Step S1
In 05, the emphasis probability Ps (e) and the quietness probability Ps obtained for each audio sub-paragraph in steps S103 and S104.
(N) and the like are sorted for each voice sub-paragraph and stored as a voice enhancement probability table in the storage means. FIG. 15 shows an example of the voice enhancement probability table stored in the storage means. Figure 15
F1, F2, F3 ... Shown in Fig. 4 are the speech sub-paragraph emphasis probability Ps (e) and the voice sub-paragraph calm probability Ps obtained for each voice sub-paragraph.
The subparagraph probability storage part which recorded (n) is shown. These audio sub-paragraphs S are stored in these sub-paragraph probability storage units F1, F2, F3, ....
Audio sub-paragraph number i attached to, start time (time counted from the beginning of the audio data string to be summarized) end time,
The audio small paragraph emphasis probability, the audio small paragraph quiet probability, the number of frames fn forming each audio small paragraph, and the like are stored.

As a condition input in the summarization condition input step S12, a summarization rate r (summarization rate described in claims) indicating that the total length of contents to be summarized is summarized in a time of 1 / X (X is a positive integer). , Which is the reciprocal of r = 1 / X), or the summary time t. For the setting of this summary condition,
In the extraction condition changing step S13, the weighting factor W is set to W = 1 as an initial value, and this weighting factor is input to the abstract extracting step S14. In the abstract extraction step S14, the weighting factor W = 1 and the emphasis probability Ps (e) and the quietness probability Ps stored for each voice sub-paragraph from the voice emphasis probability table.
(E) is compared to extract a voice sub-paragraph having a relationship of W · Ps (e)> Ps (n), and a voice paragraph including at least one of the extracted voice sub-paragraphs is extracted and extracted. The total extension time MT (minutes) of the selected voice paragraph sequence is calculated.

Total extension time MT of the extracted voice paragraph sequence
(Minutes) is compared with a predetermined summary time YT (minutes) determined by the summary condition. If MT≈YT (the error of MT with respect to YT is within a range of ± several percent, for example), the adopted audio paragraph string is reproduced as a summary audio. The error value between the summarization time YT set in the summarization condition and the summed total extension time MT of the content is larger than the stipulation, and the relationship is M.
If T> YT, the total extension time MT of the extracted audio paragraph sequence
(Minutes) is determined to be longer than the summary time YT (minutes) defined by the summary conditions, and the extraction condition changing step S1 shown in FIG.
Re-execute 3. In the extraction condition changing step S13, the total extension time MT of the voice paragraph sequence extracted when the weighting factor is W = 1.
In response to the determination result that (minutes) is “longer” than the summarization time YT (minutes) defined in the summarization conditions, the weighting coefficient W (the predetermined coefficient according to claim 1) smaller than the current value is added to the emphasis probability Ps (e). If it is larger than the current value, multiply by W · Ps
(E) Then, weighting is performed. The weighting factor W is obtained by, for example, W = 1-0.001 × K (K is the number of loops).

In other words, in the array of the enhancement probabilities Ps (e) found in all the voice sub-paragraphs of the voice paragraph sequence read from the voice enhancement probability table, W = 1−1 in the first loop.
Weighting is performed by multiplying the weighting coefficient W = 0.999 determined by 0.001 × 1. The emphasis probabilities W · Ps (e) of all the weighted voice sub-paragraphs are compared with the quietness probabilities Ps (n) of the voice sub-paragraphs, and W · Ps (e)> Ps.
An audio sub-paragraph having a relationship of (n) is extracted. In accordance with this extraction result, in the abstract extraction step S14, the voice paragraph including the extracted voice sub-paragraph is extracted, and the summary voice paragraph string is obtained again. At the same time, the total extension time MT (minutes) of this summary voice paragraph sequence is calculated, and this total extension time MT (minutes) is compared with the summary time YT (minutes) defined by the summary condition. If the comparison result is MT.apprxeq.YT, the audio paragraph string is determined as the summary audio and reproduced.

If the result of the first weighting process is still MT> YT, the extraction condition changing step is executed as a second loop. At this time, the weight coefficient W is W = 1
It is calculated by −0.001 × 2. All emphasis probabilities Ps (e)
Is weighted with W = 0.998. As described above, in this example, the extraction condition is changed such that the value of the weighting coefficient W is gradually decreased every time the loop is repeatedly executed, so that the voice amount satisfying the condition of WPs (e)> Ps (n) is reduced. You can gradually reduce the number of paragraphs. As a result, it is possible to detect the state of MT≈YT that satisfies the summary condition.

In the above description, MT≈YT is set as the convergence condition of the summarization time MT, but it is also possible to strictly set MT = YT. In this case, if the summary condition is insufficient for 5 seconds, for example, if another voice paragraph is added, it will exceed 10 seconds, but by reproducing only 5 seconds from the voice paragraph, the summary condition of the user is met. be able to. Further, the 5 seconds may be 5 seconds near the audio sub-paragraph determined to be emphasized, or 5 seconds from the beginning of the audio paragraph. When it is determined that MT <YT in the above-mentioned initial state, the weighting factor W is smaller than the current value, for example, W = 1-0.001 × K.
Then, the weighting coefficient W is multiplied by the array of the calm probability Ps (n) to weight the calm probability Ps (n). As another method, when it is determined that MT> YT in the initial state, the weighting coefficient is made larger than the current value W = 1 +
0.001 × K, and the weighting factor W is set to the calm probability Ps.
The array of (n) may be multiplied.

In the summary reproducing step S15, the audio paragraph sequence extracted in the abstract extracting step S14 is described as being reproduced. However, in the case of image information with voice, the image information corresponding to the voice paragraph extracted as the summary voice is reproduced. It is possible to summarize a television broadcast, a movie, or the like by cutting out, connecting, and playing back together with the sound. Further, in the above description, it has been described that either the emphasis probability or the quietness probability obtained for each audio sub-paragraph stored in the audio emphasis probability table is directly multiplied by the weighting coefficient W to perform weighting. For good detection, it is desirable to perform weighting as WF by multiplying the weighting coefficient W by the number F of frames forming each audio sub-paragraph.

In other words, the conditional emphasis probability Ps (e) calculated by the equations (8) and (10) is the product of the probabilities of the emphasized state obtained for each frame. Further, the probability Ps (n) of being in a calm state is also calculated by multiplying the probability of being in a calm state calculated for each frame. Therefore, for example, in order to weight the emphasis probability Ps (e), correct weighting is performed by multiplying the weighting coefficient W for each probability of the emphasis state obtained for each frame. In this case, if the number of frames forming the audio sub-paragraph is F, the weighting factor W is W
It becomes F. As a result, the influence of weighting is increased / decreased according to the number F of frames.
That is, a larger weight is given to a voice sub-paragraph having a longer extension time.

However, if it suffices to simply change the extraction condition for determining the emphasized state, the product of the probability of the emphasized state or the product of the calm state obtained for each frame is multiplied by the weighting coefficient W. However, the extraction conditions can be changed. Therefore, the weighting coefficient W does not necessarily have to be WF. Further, in the above description, the emphasis probability Ps (e) or the quietness probability P obtained for each audio sub-paragraph is used as the extraction condition changing means.
Although the method of weighting s (n) and changing the number of audio sub-paragraphs satisfying Ps (e)> Ps (n) is adopted, as another method, the emphasis probabilities Ps (e) of all audio sub-paragraphs are adopted. And the calm probability Ps (n), the probability ratio Ps (e) / P
s (n) is calculated, and the voice signal sections (voice sub-paragraphs) corresponding to the descending order of the probability ratios are accumulated to calculate the sum of the summary sections. If they match, a method of deciding the voice signal section at that time as a summary section and organizing the summary voice may be considered.

In this case, when the total extension time of the organized summary voices is excessive or insufficient with respect to the summary time set in the summary condition, the probability ratio Ps for determining the emphasized state is set.
The extraction condition can be changed by changing the threshold value for selecting the value of (e) / Ps (n). When this extraction condition changing method is adopted, it is possible to obtain an advantage that the process up to organizing the summary voice satisfying the summary condition can be simplified. In the above description, the emphasis probability Ps (e) and the quietness probability Ps (n) obtained for each audio sub-paragraph are calculated as the product of the probability of the emphasized state and the product of the probability of the calm state calculated for each frame. However, as another method, the average value of the probabilities of the emphasized state obtained for each frame is obtained, and the average value is used as the emphasis probability Ps (e) and the quietness probability Ps of the audio sub-paragraph.
It can also be used as (n).

Therefore, when the method of calculating the emphasis probability Ps (e) and the calm probability Ps (n) is adopted, the weighting coefficient W used for weighting is directly multiplied by the emphasis probability Ps (e) or the calm probability Ps (n). do it. An example of a voice processing apparatus capable of freely setting the summarization rate will be described with reference to FIG. In this embodiment, the summary condition input unit 31, the voice emphasis probability table 32, the emphasized sub-paragraph extraction unit 33, the extraction condition change unit 34, and the summary section provisional determination are added to the configuration of the voice emphasis state summarizing device shown in FIG. The unit 35, the total extension time calculation unit 35A for obtaining the total extension time of the summary voice inside the summary section provisional determination unit 35, and the total extension time calculation unit 35A.
The total extension time of the summary voice calculated by the summary condition input unit 31
The summary section determination unit 35B that determines whether or not the summary time that has been input in the setting range is included, and the summary voice storage / playback unit 35 that stores and reproduces the summary voice that matches the summary condition.
It is characterized in that it is provided with C.

As described with reference to FIG. 11, the speech feature amount of the input speech is obtained for each frame, and the emphasis probability and the quietness probability for each frame are calculated by the emphasis probability calculation unit 16 and the calm probability calculation unit 17 according to the sound characteristic amount. And are stored in the storage unit 12 together with the frame number assigned to each frame. Further, to this frame string number, the audio small paragraph string number assigned to the audio small paragraph string determined by the audio small paragraph determination unit is added, and an address is assigned to each frame and audio small paragraph. In the speech processing apparatus shown in this embodiment, the emphasis probability calculation unit 16 and the calm probability calculation unit 17 read out the emphasizing probability and the calm probability of each frame stored in the storage unit 12, and each speech is calculated from the emphasizing probability and the calm probability. The emphasis probability Ps (e) and the calm probability Ps (n) are obtained for each small paragraph, and the emphasis probability Ps (e) and the calm probability Ps (n) are stored in the voice emphasis table 32.

The voice enhancement table 32 stores the enhancement probability and the quietness probability obtained for each voice sub-paragraph of the voice waveform of various contents, and is arranged so that a summary can be executed at any time according to the user's request. There is. The user inputs the summary condition into the summary condition input unit 31. The summarization condition mentioned here indicates the name of the content to be summarized and the summarization rate for the total length of the content. As the summarization rate, an input method such as summarizing the entire length of content to 1/10 or summarizing to 10 minutes in time can be considered. Here, for example, 1
When / 10 is input, the digest time calculation unit 31A calculates a time that is 1/10 of the total time of the content, and sends the calculated digest time to the digest segment determination unit 35B of the digest segment temporary determination unit 35.

In response to the input of the abstract condition to the abstract condition input unit 31, the control unit 19 starts the operation of generating the abstract voice. As the starting work, the emphasis probability and the quietness probability of the content desired by the user are read from the voice emphasis table 32. The read emphasis probabilities and quietness probabilities are sent to the emphasis subparagraph extraction unit 33, and the voice subparagraph numbers determined to be in the emphasis state are extracted. As a method of changing the condition for extracting the voice section in the emphasized state, the above-mentioned emphasis probability Ps (e) or the quietness probability Ps (n) is multiplied by a weighting coefficient W which is the reciprocal of the probability ratio, and W · Ps ( e)> Ps
A method of extracting a voice sub-paragraph having a relationship of (n) and obtaining a summary voice by a voice paragraph including a voice sub-paragraph, and a probability ratio Ps
(E) / Ps (n) is calculated, and the probability ratio is accumulated in descending order to obtain the digest time.

As the initial value of the extraction condition, when changing the extraction condition by weighting, it is conceivable that the weighting coefficient W is set to W = 1 to be the initial value. Further, the emphasis probability Ps (e) and the quietness probability Ps (n) obtained for each audio sub-paragraph
When the emphasis state is determined according to the value of the probability ratio Ps (e) / Ps (n) of Ps (e) / Ps (n), the value of the ratio is, for example, Ps as an initial value.
It may be considered that the case where (e) / Ps (n) ≧ 1 is determined as the emphasized state. Data indicating the audio sub-paragraph number, start time, and end time determined to be emphasized in this initial setting state are output from the emphasized sub-paragraph extraction unit 33 to the summary section provisional determination unit 3
Send to 5. The summary section provisional determination unit 35 searches the voice paragraph string stored in the storage unit 12 for a voice paragraph including a sub-paragraph number determined to be in the emphasized state, and extracts the voice paragraph. The total extension time calculation unit 35A calculates the total extension time of the extracted speech paragraph sequence, and the summary section determination unit 35B compares the total extension time with the summary time input under the summary condition. If the comparison result satisfies the summarization condition, the sound paragraph sequence is stored and reproduced by the summarization sound storage / reproduction unit 35C. In this reproducing operation, an audio paragraph is extracted from the number of the audio sub-paragraph determined to be in the emphasized state by the emphasized sub-paragraph extracting unit 33, and the audio data or the video data of each content is specified by specifying the start time and the end time of the audio paragraph. Is read out and transmitted as summarized audio and summarized video data.

When the summary section determination unit 35B determines that the summarization conditions are not satisfied, the summary section determination unit 35B outputs an extraction condition change command to the extraction condition change unit 34, and the extraction condition change unit 34 outputs the extraction condition. To make changes. The extraction condition changing unit 34 changes the extraction condition and inputs the extraction condition to the emphasized small paragraph extracting unit 33. Emphasized subparagraph extraction unit 3
In accordance with the extraction condition input from the extraction condition changing unit 34, 3 again makes a comparison determination between the emphasis probability and the quietness probability of each audio subparagraph stored in the audio emphasis probability table 32. The extraction result of the emphasized small paragraph extracting unit 33 is sent again to the summary section provisional judging unit 35, and the audio paragraph including the audio small paragraph judged to be emphasized is extracted. The total extension time of the extracted voice paragraph is calculated, and the summary section determination unit 35B determines whether or not the calculation result satisfies the summary condition. This operation is repeated until the summarization condition is satisfied, and the audio paragraph string satisfying the summarization condition is read from the storage unit 12 as the summarized audio and summarized video data and distributed to the user terminal.

In the above description, the start time and the end time of the summary section are extracted as the start time and the end time of the audio paragraph string that is determined to be the summary section, but it is determined to be the summary section in the case of the content with video. Detects the cut point of the video signal near the start time and end time of the audio paragraph sequence,
A method of defining the start time and the end time of the summary section by the time of this cut point (for example, as described in Japanese Patent Application Laid-Open No. 8-32924, a signal generated when switching the detected screen is used) can be considered. In this way, when the cut point of the video signal is used for the start time and end time of the summary section,
Since the switching of the summary section is synchronized with the switching of the image,
There is an advantage that the visibility is improved visually and the comprehension of the summary can be improved.

As described above, since the voice summarization method used in the present invention extracts a voice section having a high probability of being in the emphasized state of the voice waveform as a summary section, it is extracted at the same start time and end time as this voice section. In many cases, the video information to be displayed is also an important video part in the content. As a result,
According to the summarization method used in the present invention, it is possible to obtain the advantage that the summarization information that can appropriately convey the content of the content to the viewer can be obtained. FIG. 1 shows an example of the processing procedure of the video reproducing method according to the present invention. In the present invention, a real-time video signal and an audio signal being broadcast are recorded in association with a reproduction time, and the recorded portion is summarized and reproduced while maintaining this recording state. Hereinafter, this situation will be referred to as catch-up reproduction.

In step S111, the catch-up reproduction target start time or the catch-up reproduction target start video is specified. For this identification, for example, when the viewer leaves the seat temporarily while watching the program, the leaving time is designated by a push button operation. Alternatively, a sensor is provided on the door of the room, and when the door is opened or closed, it is detected that the viewer has left the room, and the leaving time is designated. Alternatively, a method may be considered in which, regardless of whether or not the user is away from the seat, a part of a program that has already been recorded is fast-forwarded and played back, and the viewer arbitrarily specifies the start position of catch-up playback from the video. In step S112, the summary condition (summarization time or summary rate) is input. This input is made when the viewer who leaves the seat returns to the seat. That is, when the time during leaving the seat is, for example, 30 minutes, the viewer inputs the summary condition according to the viewer's idea as to how much the content broadcast in the 30 minutes should be compressed and viewed. As another input method, if there is no input from the viewer, a preset default value, for example, 3 minutes is used, or some candidates are prepared and displayed, and the viewer instructs A method of selecting and inputting is conceivable.

Further, for example, when the viewer returns home while the recording is automatically started by the reserved recording, since the recording start time is known by the reserved recording, the digest end time is determined by the digest reproduction start instruction. To be done. Here, for example, if the summarization condition is determined in advance by a default value or the like, the recording start time to the summarization end time are summarized according to the summarization condition. In step S113, the catch-up reproduction start is designated. By specifying this catch-up playback start,
The end point (summary end time) of the section to be summarized is specified.
As an input method, the viewer specifies the operation time by a push button operation to instruct the start of catch-up playback, or the viewer's entry into the room is detected by an opening / closing sensor installed in the room door, and catch-up playback is performed with the entry time. You may specify the start.

In step S114, the reproduction of the video currently being broadcast is stopped. In step S115, summary processing, summary video /
Play audio. The summarization process specifies a summarization section according to the summarization condition specified for the catch-up reproduction target start time or the catch-up reproduction target start video defined in step S111 to the summarization end time input in step S113, and identifies the summarization section. An audio signal and a video signal synchronized with the summary section are reproduced. In step S116, the reproduction of the summary section ends. In step S117, the video reproduction during broadcasting is reunited.

FIG. 2 shows an example of the video reproducing apparatus 100 of the present invention which executes the above-mentioned catch-up reproduction. The video reproducing apparatus 100 according to the present invention includes a recording unit 101, an audio separating unit 102, an audio summarizing unit 103, and a summarizing section reading unit 10.
4 and the mode switching unit 105. The recording unit 101 uses recording / reproducing means such as a hard disk, a semiconductor memory, or a DVD-ROM that can perform writing and reading at high speed. Since writing and reading can be performed at high speed, it is possible to reproduce a portion that has already been recorded while recording a program currently being broadcast. The input signal S1 is input from a television tuner or the like and may be either an analog signal or a digital signal. However, the recording of the recording unit 101 is performed by a digital signal.

The audio separating unit 102 separates audio from the video signal in the specified summarization target section, and inputs this audio signal to the audio summarizing unit 103. The voice summarization unit 103 extracts the emphasized portion using this voice signal and specifies the summarization section.
Note that the audio summarization unit 103 always analyzes the audio signal during recording, creates the audio enhancement table shown in FIG. 15 for each recorded program, and stores it in the storage unit. Therefore, when the summary reproduction of the recorded portion is performed midway during the broadcasting of the program, the summary is performed using the voice emphasis table. Further, when viewing the summary of the recorded program at a later date, the summary is performed using the voice emphasis probability table.

The summary section reading unit 104 is a voice summarizing unit 103.
The video signal with audio is read from the recording unit 101 in accordance with the summary section specified in 1. and output to the mode switching unit 105.
The mode switching unit 105 outputs the video signal with audio read by the summary section reading unit 104 as a summary video signal and provides it for viewing by a viewer. The mode switching unit 105 is switched to a reproduction mode b for outputting the video signal read from the recording unit 101 and a mode c for directly providing the input signal S1 for viewing, in addition to the mode a for outputting the summary video, and various modes. It is available in the form. By the way, in the catch-up reproduction method described above, since the video broadcast during the execution period of the catch-up reproduction is not included in the summary target section, the viewer cannot view the video.

Therefore, the present invention proposes the following embodiments in order to solve this drawback. That is, after the reproduction of the summary section is completed, the previous reproduction start time is set as a new start time of the summary target, and the summary section reproduction end time is set as the new summary end time, and the summary processing and the reproduction processing of the summary video / audio are repeated. When the time between the previous reproduction start time and the summary section reproduction end time is equal to or less than a predetermined time (for example, 5 to 10 seconds), the repetition is ended. In this case, there arises a problem that the summary section is reproduced longer than the specified summarization rate or summarization time. For example, when the time of the section to be summarized is T _A , the summary rate r (0 <r <1, r = summary time (total extension of time of the summary section) / time of the section to be summarized) is summarized as follows.
Summary time T ₁ in the first round of the summary will be T _A r. Summary time in the second abstract becomes T _A r ² so further summarized summarization rate for those of the first. Repetition time required until completion This process is repeated sequentially is T _A r
/ (1-r).

The summarization rate r specified here is r / (1+
r) and perform the summarization with this adjusted summarization rate.
U In that case, the time required to complete the repetition is T_Ar
The summarization time is suitable for the specified summarization rate. As well
To the summary time T₁Even when is specified, if the section is a summary target
Interval T_AIs given, the specified summarization rate r is T₁/
T_ATherefore, the summarization rate is T₁/ (T_A+ T₁) As
Also the first summary time T _AT₁/ (T_A+ T₁)
Good. FIG. 3 shows another embodiment for solving the above-mentioned inconvenience.
Indicates. In this embodiment, the input signal S1 is output as it is.
The main screen 200 of the display unit (see FIG. 4) is currently being broadcast.
Along with displaying the image, the mode switching unit 105 displays a sub screen.
A sub-screen processing summary video / sound provided with the conversion processing unit 106
The voice is superimposed on the input signal S1 and output, and this summary video is displayed.
The mixed mode d to be displayed on the child screen 201 (see FIG. 4)
The provided example is shown.

According to this embodiment, the viewer can view the main screen 20.
While viewing the content of the program being broadcast that is displayed at 0, the content of the program that was broadcast in the past can be summarized and viewed on the child screen 201. As a result, the content of the program being broadcast while viewing the summary information can be received from the main screen 200, so that when the summary information is completely reproduced, the content of the program is currently being broadcast from the first half. It is possible to make the viewer understand the content up to the point of time almost seamlessly. The video reproducing method according to the present invention described above is realized by causing a computer to execute a video reproducing program.
In this case, the video reproduction program is downloaded via a communication line, or is stored in a recording medium such as a CD-ROM or a magnetic disk and installed in a computer, and the processing device such as a CPU in the computer executes the method of the present invention. Let

[0065]

As described above, according to the present invention, a recorded program is compressed to an arbitrary compression rate and summarized.
This summarized summary information can be reproduced. Therefore, the contents of a large number of recorded programs can be distinguished in a short time, and, for example, the work of searching for a program to be viewed can be completed in a short time. In addition, even if a program that was not available from the beginning is being broadcast, the first half of the program can be summarized and viewed, so even if the program was started halfway, the entire program can be grasped. You can continue watching.

[Brief description of drawings]

FIG. 1 is a flowchart for explaining an example of a procedure for executing a video reproducing method according to the present invention.

FIG. 2 is a block diagram for explaining an embodiment of a video reproducing apparatus according to the present invention.

FIG. 3 is a block diagram for explaining a modified embodiment of the video reproducing apparatus according to the present invention.

FIG. 4 is a diagram for explaining an example of a display screen for explaining the features of the video reproducing apparatus according to the present invention.

FIG. 5 is a flowchart for explaining the voice summarization method proposed above.

FIG. 6 is a flowchart for explaining a previously proposed method of extracting a voice paragraph.

FIG. 7 is a diagram for explaining the relationship between audio paragraphs and audio subparagraphs.

8 is a flowchart showing an example of a method of determining the utterance state of an input voice sub-paragraph in step S2 shown in FIG.

FIG. 9 is a flowchart showing an example of a procedure for creating a codebook used in the previously proposed speech summarization method.

FIG. 10 is a diagram showing a storage example of a codebook used in the present invention.

FIG. 11 is a waveform diagram for explaining a speech state likelihood calculation.

FIG. 12 is a block diagram for explaining an embodiment of the previously proposed voice enhancement state determination device and voice summarization device.

FIG. 13 is a flowchart for explaining a summarization method in which the summarization rate can be changed freely.

FIG. 14 is a flowchart for explaining an operation of extracting a voice sub-paragraph used for summarizing a voice, an operation of calculating an emphasis probability of each voice sub-paragraph, and an operation of extracting a voice sub-paragraph calm probability.

FIG. 15 is a diagram for explaining the configuration of a speech emphasis probability table used in the speech summarization device.

FIG. 16 is a block diagram for explaining an example of a voice summarizing device capable of freely changing the summarization rate.

[Explanation of symbols]

100 Video Playback Device 101 Recording Unit 102 Audio Separation Unit 103 Audio Summarization Unit 104 Summary Section Reading Unit 105 Mode Switching Unit 106 Child Screening Processing Unit

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G10L 15/06 H04N 5/91 C 15/10 G10L 3/00 531N H04N 5/91 521U 513A (72) Invention Person Mizuno Osamu, Otemachi 2-3-1, Chiyoda-ku, Tokyo F-Term (Reference) within Nippon Telegraph and Telephone Corporation 5C053 HA27 JA23 LA11 5D015 DD03 FF00 FF05 HH04 KK01

Claims (10)

[Claims] 1. A real-time video signal and an audio signal are stored in association with a reproduction time, a summary start time is input, and a summary time that is the total extension time of the summary section or all summary objects of the total extension time of the summary section. By inputting a summarization rate which is a ratio of sections, the section determined to be the emphasized state for the speech signal in the section to be summarized up to the present as the summarization start time to the summarization end time in the summarization time or summarization rate is determined to be the summarization section. A video reproducing method characterized by reproducing the audio signal and the video signal in the summary section. 2. The summary section is determined such that the reproduction end time of the audio signal and the video signal of the summary section is a new summary section end time and the reproduction end time of the summary section is a new summary reproduction start time. The method according to claim 1, wherein the reproduction of the audio signal and the video signal in the summary section is repeated. 3. The summarization rate r (r is a real number satisfying 0 <r <1) is adjusted to r / (1 + r), and the summarization section is determined based on the adjusted summarization rate. Two
Video playback method described. 4. At least a fundamental frequency or pitch period,
Power, a time-varying characteristic of a dynamic feature amount, or using a codebook that stores the feature amount including these inter-frame differences, the appearance probability in the emphasized state, and the appearance probability in the quiet state in association with each other, The appearance probability in the emphasized state corresponding to the feature amount analyzed for each frame is obtained, the appearance probability in the static state corresponding to the feature amount analyzed for each frame of the audio signal is obtained, and the appearance in the emphasized state is obtained. The probability of becoming an emphasized state is calculated based on the probability, the probability of becoming a calm state is calculated based on the appearance probability in the calm state, and the probability ratio of the probability of becoming the emphasized state to the probability of becoming the calm state is voiced. It is calculated for each signal section, and the summarization time is calculated by accumulating the time of the voice signal section corresponding to the probability ratio in descending order, and the summarization rate, which is the ratio of the summarization time to all summarization target sections, is input. 4. The video reproducing method according to claim 1, wherein an audio signal section having a reduction rate is determined as the summary section. 5. At least a fundamental frequency or pitch period,
Power, a time-varying characteristic of a dynamic feature amount, or using a codebook that stores the feature amount including these inter-frame differences, the appearance probability in the emphasized state, and the appearance probability in the quiet state in association with each other, The appearance probability in the emphasized state corresponding to the feature amount analyzed for each frame is obtained, the appearance probability in the static state corresponding to the feature amount analyzed for each frame of the audio signal is obtained, and the appearance in the emphasized state is obtained. The probability of being in an emphasized state is calculated based on the probability, the probability of being in a stationary state is calculated based on the appearance probability in the stationary state, and the probability ratio of the probability of becoming the emphasized state to the probability of becoming the stationary state is predetermined. A voice signal section larger than the coefficient of tentatively is determined to be a summary section, and the sum of the time of the summary section or a ratio of the time of all the voice signal sections to the sum of the time of the summary section is calculated as a summarization rate, 4. The summary section is determined by calculating the predetermined coefficient such that the sum of the summarization time periods is approximately a predetermined summarization time or the summarization rate is a substantially predetermined summarization rate. The video reproduction method according to any one of 1. 6. It is determined whether or not the voice signal is in a silent section for each frame, and whether or not it is a voiced section, and a portion surrounded by a voiceless section of a predetermined number of frames or more and a section including a voiced section is determined as a voice sub-paragraph, The average power of the voiced section included in the audio sub-paragraph is determined to be a voice sub-paragraph group ending with a voice sub-paragraph that is smaller than a predetermined constant multiple of the average power in the voice sub-paragraph, and the voice signal section is The video reproduction method according to claim 4, wherein the video reproduction method is defined for each audio paragraph, and the summary time is accumulated for each audio paragraph. 7. Storage means for storing a real-time video signal and audio signal in association with a reproduction time, a summary start time input means for inputting a summary start time, and a summary time or a summary which is a total extension time of the summary section. Summary condition input means for inputting a summary condition defined by a summarization rate that is a ratio of all summary target sections of the total extension time of the section, and according to the summary condition, the summary target from the summary start time to the present as the summary end time A summarization section determining unit that determines a section that is determined to be an emphasized state for an audio signal in the section as a summary section; and a reproducing unit that reproduces the audio signal and the video signal of the summary section determined by the summary section determining unit. Video playback device characterized by. 8. The video reproducing apparatus according to claim 7,
The summarization section determination means determines at least the fundamental frequency or pitch period, the power, the time change characteristic of the dynamic feature amount, or the feature amount including the difference between the frames, the appearance probability in the emphasized state, and the appearance probability in the quiet state. Correspondingly stored codebook, using the codebook to obtain the appearance probability in the emphasized state corresponding to the feature amount of the voice signal analyzed for each frame, and emphasized based on the appearance probability in the emphasized state An emphasis state probability calculation unit that calculates the probability of being in a state, and obtains an appearance probability in a quiet state corresponding to the feature amount obtained by analyzing the voice signal for each frame using the codebook,
A calm state probability calculating unit that calculates a probability of being in a calm state based on the appearance probability in the calm state, and a probability ratio of the probability of being the emphasized state to the probability of being the calm state is calculated for each voice signal section. A summarization time is calculated in a descending order by accumulating the time of the voice signal section corresponding to the probability ratio, and a summarization section provisional determination unit that provisionally determines a summarization section; and a ratio of the summarization section to all summarization target sections is the summarization rate. A video reproduction device comprising: a digest section determining unit that determines an audio signal section that satisfies the above as the digest section. 9. The video reproducing apparatus according to claim 7,
The summarization section determination means determines at least the fundamental frequency or pitch period, the power, the time change characteristic of the dynamic feature amount, or the feature amount including the difference between the frames, the appearance probability in the emphasized state, and the appearance probability in the quiet state. Correspondingly stored codebooks, using this codebook, the speech code is analyzed for each frame, and the appearance probability in the emphasized state and the appearance probability in the quiet state corresponding to the feature amount are obtained. An emphasized state probability calculation unit that calculates the probability of becoming an emphasized state based on the appearance probability of, a calm state probability calculation unit that calculates the probability of becoming a calm state based on the occurrence probability in the calm state, and the emphasized state And a probability ratio of the probability of being in the quiet state to a speech signal section having a larger coefficient than a predetermined coefficient. And a summarization section determining unit that determines the summarization section for each channel or each speaker by calculating the predetermined coefficient at which the summarization rate becomes a substantially predetermined summarization rate. Video playback device. 10. A video reproduction program which is described by a computer readable code and executes the video reproduction method according to claim 1. Description: