JP5182892B2 - Voice search method, voice search device, and voice search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To search an optional audio section of a video, an audio and music contents by a reference audio section, and to thereby acquire a scene having a similar mental impression in both audio sections as a search result. <P>SOLUTION: A method includes the steps of: dividing the audio by an audioless section (S1); extracting rhythm characteristics related to a fundamental frequency of the divided audio section or the degree of emphasis of the audio (S2); and calculating a distance between the rhythm characteristics of the reference audio section and the rhythm characteristics of one or more audio sections which are investigation objects (S3). An audio section having similarity equal to or higher than a prescribed value is estimated by the calculated distance (S4). <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a voice search method, device, and program for searching a voice section.

  If a scene that is impressively similar to an arbitrary voice section can be detected, it can be applied to a user's favorite scene search, which is highly convenient. For example, Non-Patent Document 1 shows a technique for retrieving a similar scene from a time history of a fundamental frequency, which is a known technique.

  In addition, as a method for acquiring voice sensitivity information, Patent Document 1 discloses a technique for extracting a voice enhancement state.

  In addition, according to the technique described in Non-Patent Document 2, a technique for generating a shortened video image by paying attention to voice laughter is shown.

  Further, in Patent Document 2, when searching a video / audio database having a high similarity to a query, the emotion / emotion level is estimated for the video / audio in the video / audio database. (Word) or emotion level is used as a query, and a technique for searching for a high emotion (word) or emotion level similarity is described.

Japanese Patent No. 3803311 JP 2008-204193 A

Isamu Saito, Nobuya Nakajo, “Attempt of Kansei Information Retrieval Interface-Content Retrieval by Speech Prosodic Information”, Proceedings of the 36th Annual Conference of the Institute of Image Electronics Engineers of Japan, R.3-5, 2008 Go Irie, Kota Hidaka, Naoya Miyashita, Takashi Sato, Yukinobu Taniguchi, “Laughter” Scene Detection Method for Video Quick Reference for Personal Video, Journal of the Institute of Image Information and Television Engineers, vol.62, no .2, pp.227-233, 2008.

  However, since the method of Non-Patent Document 1 retrieves from the time history of the fundamental frequency, for example, the influence of the fundamental frequency extraction error such as double pitch and half pitch cannot be ignored. As a result, it was not possible to detect a desired speech segment.

  In addition, the methods of Patent Document 1 and Non-Patent Document 2 show video, audio, and music summarization techniques, and do not indicate that they are used for searching.

  Patent Document 2 discloses an invention for searching video and audio by using a category of emotion (emotion word) such as “fun” and “sad” and an emotion level as a query. In the present invention, the emotion word and the emotion level induced by the video and audio in the database are estimated. The user can search for video / sounds with similar impressions by giving the emotional word and emotion level of the video / sound that he / she wants to watch as a query. However, since emotions are highly subjective and the boundaries between categories are ambiguous, the user may not always be able to express a query as an appropriate emotion word or emotion level. In addition, there is a problem that it is not always possible to present the search result assumed by the user because similarity is judged by the emotion word and the emotion level accompanied by fluctuation of subjective influence.

  The present invention has been made paying attention to the above circumstances, and the object of the present invention is to search an arbitrary voice section of video, voice, and music content by a reference voice section and to obtain a psychological result of both voice sections as a search result. The object is to provide a technology that makes it possible to obtain a similar impression.

  In order to achieve the above object, the present invention focuses on the prosodic feature as a feature amount indicating the impression that the voice gives to the viewer, and calculates the distance of the prosodic feature for each voice section, thereby giving the impression given to the person. Extracting similar speech segments, inputting speech as a query, extracting prosodic features, and determining audio / video segments with similar prosodic features as speech segments with similar impressions to people It is a feature. By using voice as a query, the user can save time and effort to express the impression in words. In addition, by using speech as a query and performing similarity determination using feature quantities, it is possible to avoid mixing subjective fluctuations caused by expressing an impression in words.

  That is, the present invention comprises the following means.

The first means extracts the voice from the video, further divides it into voice sections, calculates the distance from the reference voice section based on the prosodic features, and estimates a similar voice section. In addition, the fundamental frequency is further calculated by the weighted average fundamental frequency, and is converted into semitones to obtain prosodic features. According to this first means, it is possible to search for a speech segment similar to the psychological impression of the reference speech segment. Further, the fundamental frequency can be obtained by reducing the influence of double pitch and half pitch.

The second means calculates the distance between both speech sections by the dynamic time expansion / contraction method. According to the second means, even when two or more speech sections to be compared have different time lengths, the similarity between the speech sections can be obtained.

  In order to solve the problem of Patent Document 2, the present invention uses voice signals directly as queries instead of emotions (words) and determines similarity at the feature level. Furthermore, similarity is also determined at the feature level.

  According to the present invention, it is possible to search for a speech segment similar to the psychological impression of the reference speech segment. Further, since the voice signal is directly used as a query, a case where the user cannot express the query can be avoided. Furthermore, since the similarity is also determined at the feature amount level, it is not easily affected by fluctuations in subjectivity, and the user's intention with respect to the search result is less likely to occur.

It is a processing flowchart which shows an example of the voice search method by this invention. It is a figure explaining the example of the weighted average of a fundamental frequency. It is a figure explaining the example which semitone-converts a fundamental frequency. It is a figure which shows the structural example of the speech search device by this invention. It is a figure which shows the data structural example of a content memory | storage part and a search object audio | voice area prosodic feature memory | storage part. It is a figure which shows the usage example of a voice search device. It is a figure which shows the example of the user display screen in the case of an automatic search.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a processing flow of a voice search method according to an embodiment of the present invention. Step S1 is a voice division step, in which the input voice is divided into arbitrary voice sections. Step S2 is a prosodic feature extraction step, which extracts prosodic features. Step S3 is a distance calculation step, in which the similarity between the prosodic feature of the speech section and the prosodic feature of the predetermined speech section is obtained as a distance. Step S4 is a speech segment estimation step, in which it is determined whether or not they are similar.

  First, in the voice division step S1, the input voice is divided into voice sections by voiced / unvoiced voice judgment. For example, a frame from which a fundamental frequency is not extracted may be unvoiced, or a threshold value may be set for the correlation coefficient of the prediction residual of LPC analysis, and the threshold value or less may be unvoiced.

In the prosodic feature extraction step S2, the fundamental frequency is extracted by any one of waveform processing, correlation processing, and spectrum processing. Normally, prosodic features such as the fundamental frequency are extracted every short time interval (called a window) of several tens of milliseconds. This window is an analysis frame, and in the prosodic feature extraction in this step S2, extraction may be performed at intervals of the analysis frame width such as every 10 ms (milliseconds) or every 30 ms. The LPC analysis method and the fundamental frequency extraction method are described in Reference Document 1 below, and are known techniques, and thus a detailed description thereof is omitted here.
[Reference 1] Sadahiro Furui, Digital Signal Processing, Tokai University Press, pp. 57-59.
The accent of the Japanese word may be given one value for each beat (mora), that is, it may be a prosodic feature by this point pitch. Alternatively, for simplicity, for example, a weighted average F0 with an amplitude of 50 ms may be used as the point pitch. The amplitude of 50 ms is not limited and may be 100 ms, for example.

  FIG. 2 is a diagram illustrating an example of a weighted average of fundamental frequencies. As shown in FIG. 2, it is assumed that both the fundamental frequency (F0) and the amplitude are extracted at intervals of 10 ms. Then, five sets of fundamental frequencies and amplitudes (F1, p1) to (F5, p5) are obtained in 50 ms. In the average of normal fundamental frequencies, “(F1 + F2 + F3 + F4 + F5) / 5” is an average value, but in the case of weighted average based on amplitude, “(α1 × F1 + α2 × F2 + α3 × F3 + α4 × F4 + α5 × F5) / 5” is basic. It is a weighted average of frequencies. Here, αi is a value obtained by normalizing pi to take a value of 0 to 1.

  Further, the fundamental frequency obtained as described above may be semitoned to obtain a prosodic feature. FIG. 3 is a diagram for explaining an example in which the fundamental frequency is made semitone.

  Semitoning is a process of converting from a fundamental frequency to a scale (semitone) C, C #,. Semi-toning is introduced for the purpose of eliminating the problems caused by double pitch (a case where a frequency twice the original fundamental frequency is extracted) and half pitch (a case where a half frequency is extracted). One octave is just a double of the fundamental frequency, and double pitch and half pitch are classified into the same scale. Therefore, the effect of the above-mentioned problem can be avoided by semitone.

  As an example of specific semitone processing, for example, a semitone / basic frequency correspondence table shown in FIG. 3 is prepared in a storage device in advance. When the fundamental frequency is extracted, the extracted fundamental frequency is converted into a semitone with reference to the semitone / basic frequency correspondence table shown in FIG. In this case, for example, it may be classified into the smallest semitone as the square distance. Usually, the fundamental frequency of human voice is said to be about 75HZ to 450Hz. Therefore, when the fundamental frequency is lower or higher than the fundamental frequency in the table of FIG. 3, the fundamental frequency set in the table is the octave principle. In accordance with the above, measure the distance to double or halve and select the appropriate semitone.

  Also, the enhancement degree may be obtained from the speech enhancement probability and the calm probability, and this may be used as a prosodic feature. Specifically, the probability ratio between the emphasis probability Ps (e) and the calm probability Ps (n) for each audio sub-paragraph may be Ps (e) / Ps (n). What is necessary is just to obtain | require the said emphasis probability and calm probability by the method of patent document 1. FIG.

  In the distance calculation step S3, when comparing two or more speech sections, the comparison is not easy because the time lengths of the sections are different. Therefore, the distance is obtained by the dynamic time expansion / contraction method. The dynamic time expansion / contraction method is described in the above-mentioned Reference 1 (written by Sadahiro Furui, Digital Signal Processing, Tokai University Press, pp. 162-164).

  In the speech section estimation step S4, it is only necessary to output a speech section in which the distance obtained as d is the minimum. In the present invention, the input speech segment is referred to as a reference speech segment, and the speech segment to be searched is referred to as a survey target speech segment. In the above processing, the prosodic features are read from the storage unit storing the prosodic features of one or more speech sections to be investigated, and the distance from the prosodic features of the reference speech sections obtained in the prosodic feature extracting step S2 is calculated by distance calculation. Obtained in step S3. In the speech segment estimation step S4, the speech segment with the closest distance, that is, the speech with a high degree of similarity is output.

  According to the present invention, two or more speech segments having similar prosodic features have an effect of being approximated also in a psychological impression scale. For example, a user who is interested in an arbitrary section of a movie or television broadcast program can search for a section similar to the section.

  FIG. 4 is a diagram showing a configuration example of a voice search device that executes the above processing. As shown in FIG. 4, the voice search device 10 includes a central processing unit (CPU) 11. A program memory 13, a data memory 14, and a communication interface (communication I / F) 15 are connected to the CPU 11 via a bus 12.

  The program memory 13 stores a program for realizing the speech dividing unit 13a, the prosodic feature extracting unit 13b, the distance calculating unit 13c, and the speech segment estimating unit 13d. The data memory 14 is provided with a content storage unit 14a and a survey target speech segment prosodic feature storage unit 14b. The speech dividing unit 13a, the prosody feature extracting unit 13b, the distance calculating unit 13c, and the speech segment estimating unit 13d are respectively the speech segmentation step S1, the prosody feature extraction step S2, the distance calculation step S3, and the speech segment estimation step S4 in FIG. Supports processing functions.

  The voice search device 10 shown in FIG. 4 searches a large amount of content (video / sound / music) that is close to the content input by the user as an inquiry in terms of prosodic features. The content storage unit 14a is a so-called content database, in which a large amount of content is stored. Also, the time history of the prosodic features is stored for each speech segment of the content in the survey target speech segment prosodic feature storage unit 14b.

  FIG. 5 shows a configuration example of data stored in the content storage unit 14a and the survey target speech segment prosodic feature storage unit 14b shown in FIG.

  A201 illustrated in FIG. 5 is a content table of the content storage unit 14a. Here, a file path indicating the location of the content itself (file) and a content ID for uniquely identifying these are stored in association with each other.

  a202 is an audio section table of content ID = 0001 in the content storage unit 14a. In this example, an audio section table is prepared for each content ID. Since one content has a plurality of audio sections, the audio section ID for uniquely identifying them, the file path indicating the location of the audio file in the section, and the contents in which each audio section exists Are stored in the speech section table a202 in association with each other.

  a203 is an example of prosodic feature data stored in the investigation target speech section prosodic feature storage unit 14b, and indicates the prosodic features of content ID = 0001 and speech section ID = s0001. As described in the prosodic feature extraction step S2 in FIG. 1, prosodic features are extracted in units of 50 ms, for example, so that a plurality of prosodic features are extracted every 50 ms for one speech section. Therefore, as indicated by a203 in FIG. 5, the fundamental frequency is stored as a prosodic feature every 50 ms for the speech section ID = s0001.

  Each of these table data can be cross-referenced like a relational database. Therefore, after determining the similarity of the speech segment by the prosodic feature of a203, this segment is in which speech segment (ID) of which content (ID). You can pull out what it is.

  A case where a user is interested in a scene and searches for a similar scene will be described. The audio dividing unit 13a in FIG. 4 performs the same processing as the audio dividing step S1 after extracting the audio when the medium is video. Here, even in the case of music, processing similar to that for voice is performed. Therefore, in the present invention, the term “speech” includes all musical sounds such as music.

  The designated scene is stored in the content storage unit 14a, and the speech information divided by the speech dividing unit 13a and the prosodic features of the speech section are extracted by the prosodic feature extracting unit 13b, and the target speech section prosodic feature storing unit 14b is extracted. The distance from the prosodic feature is obtained by the distance calculation unit 13c, and the speech section having the minimum distance is identified by the speech section estimation unit 13d. Then, a scene synchronized with the same voice section is extracted from the content storage unit 14a and presented to the user.

For example, as shown in FIG. 6, a search request may be sent from the personal computer (PC) of the user terminal 20 to the voice search device 10 via a network or the like. In that case, the search request is
・ Media such as video and audio,
・ Time information of media such as video and audio,
It's okay.

  Alternatively, in the case where the user terminal is the voice search device 10 shown in FIG.

  The similar scene search may be automatically performed without waiting for the user search. For example, the prosodic feature and the emotional state of the investigation target speech section are stored in advance in the investigation target speech section prosodic feature storage unit 14b. In that case, it may be associated with emotions such as fun, happiness, anger, disgust, giving up, sadness, calmness and surprise. Furthermore, for example, if a fun level is assigned, it is possible to automatically estimate that the scene is a more enjoyable scene from the prosodic features of the more enjoyable scene. For example, when a “pretty fun” scene is displayed on the display terminal of the user, a “pretty fun” scene of another scene may be automatically inserted as shown in the user display screen of FIG. If the user is interested in the inserted scene, the user may take measures such as switching to viewing the inserted scene. In this way, a device that automatically recommends similar scenes is realized.

  When storing the prosodic features and the emotional state of the survey target speech section in the survey target speech segment prosodic feature storage unit 14b in association with each other, it is not always necessary to detect the sensitivity of each speech segment of all contents. It is not necessary to associate the states with each other, and a model (table) in which a time history pattern of a small group of prosodic features and the emotional state are associated with each other may be prepared manually by input from an input device. After the input of this model, the Kansei state can be automatically given to the remaining speech sections with reference to this model. As the model, for example, HMM (Hidden Markov Model) can be used.

Assuming that the above process created a model that correlates the time history pattern of prosodic features with the emotional state,
(1) Extract the prosodic features of the speech segment of the content that the user is viewing,
(2) By using the model, acquire the sensitivity state of the extracted prosodic features,
(3) By using the process of searching the content storage unit 14a for the content corresponding to the obtained sensitivity state, the use as shown in FIG. 7 can be realized.

  The above voice search processing can be realized by a computer and a software program, and the program can be recorded on a computer-readable recording medium or provided through a network.

10 Voice Retrieval Device 11 Central Processing Unit (CPU)
12 bus 13 program memory 13a speech division unit 13b prosodic feature extraction unit 13c distance calculation unit 13d speech segment estimation unit 14 data memory 14a content storage unit 14b survey target speech segment prosody feature storage unit 15 communication interface

Claims (5)

In a voice search method in which a computer searches a voice segment,
A voice division step of dividing the voice into a plurality of voice sections by a silent section;
A prosodic feature extracting step of extracting prosodic features in units of analysis frames for the speech section;
One or more of the probing features stored in the probing feature prosodic feature storage unit that stores the prosodic features extracted in advance in the prosodic feature extracting step of the reference speech segment and the prosodic features for each speech segment of the target speech A distance calculating step for calculating a distance from the prosodic feature of the speech section of
Performing a speech segment estimation step for estimating a speech segment having a similarity equal to or greater than a predetermined value based on the calculated distance ;
The prosody feature extraction step includes a weighted average fundamental frequency extraction step for extracting a fundamental frequency by a weighted average or a point pitch, and a semitone process for semitoning the fundamental frequency. A voice search method characterized by using a fundamental frequency . In the audio retrieval method according to claim 1 Symbol placement,
In the distance calculating step, the distance is calculated based on a dynamic time expansion / contraction method. In a voice search device that searches a voice segment,
A voice dividing unit that divides the voice into a plurality of voice segments by a silent segment;
A prosodic feature extraction unit that extracts prosodic features in units of analysis frames for the speech section;
A survey target speech segment prosody feature storage unit that stores a prosody feature for each speech segment of the survey target speech in advance;
Distance calculation for calculating a distance between the prosodic feature extracted by the prosodic feature extraction unit of the reference speech segment and the prosodic feature of one or more speech segments to be investigated stored in the surveyed speech segment prosodic feature storage unit Part,
A speech segment estimation unit that estimates a speech segment having a similarity greater than or equal to a predetermined value based on the calculated distance ;
The prosody feature extraction unit includes a weighted average fundamental frequency extraction unit that extracts a fundamental frequency by weighted average or point pitch, and a semitone unit that semitones the fundamental frequency, and semitones the prosody feature. A voice search device characterized by having a fundamental frequency . In speech retrieval apparatus according to claim 3 Symbol mounting,
The distance calculation unit calculates a distance based on a dynamic time expansion / contraction method. A voice search program for causing a computer to execute the voice search method according to claim 1 .

Images