JP5296598B2 - Voice information extraction device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a voice information extraction device for displaying the voice recognition result of a program or the like and a topic related with speech content as a retrieval result. <P>SOLUTION: A voice information extraction device includes: a video voice recording part for obtaining a video and a voice; a voice recognition part for performing the voice recognition processing of a voice by using an acoustic model and a language model; a text data obtaining part for obtaining text data related with the obtained video and voice from the outside; a topic extraction part for extracting a topic by comparing the obtained text data with the result of the voice recognition processing; a voice information integration part for writing voice information configured by integrating the voice recognition result with the topic in the voice information storage part; an index creation part for retrieval for creating an index for retrieval on the basis of the voice recognition result; and a retrieval server part for retrieving index voice information for retrieval on the basis of a retrieval request on the basis of a retrieval word, and for reading and presenting the voice information related with the pertinent video and the voice. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an audio information extraction apparatus that extracts audio information from input video / audio, and makes it possible to search and browse the extracted audio information.

  In Patent Document 1, in particular, the video signal included in the broadcast program data of the television is displayed on the display device, and voice recognition processing is performed on the audio data included in the broadcast program data. Acquires text information, extracts keywords by morphological analysis of the acquired text information at a predetermined time, stores the extracted keywords and the information of the time together in a storage device, and stores them as a history. There is described an information processing apparatus that displays a list of keywords stored in the storage device together with selected time information when any time information is selected. Further, by using this keyword, it becomes possible to acquire related detailed information from the outside via a communication line such as the Internet.

Non-Patent Document 1 and Non-Patent Document 2 describe techniques for expanding and compressing lattice data of speech recognition results.
The method described in Non-Patent Document 1 examines the similarity between the utterance time overlaps and the pronunciation of the word notation when the lattices are aggregated. For example, “Lincoln” and “Inkan” are checked for similar pronunciation. As a result, it is possible to obtain an opposing relationship between word hypotheses (correct answer candidates).
In the method described in Non-Patent Document 2, when a lattice is compressed, a graph (maximum likelihood sequence) based on the compression lattice is selected in advance from the lattice. Then, nodes and edges are added to the compressed lattice while changing the order of circulating the lattice.

JP 2009-077166 A

L. Mangu, E. Brill, A. Stolcke, “Finding consensus in speech recognition: word error minimization and other applications of confusion networks”, Computer Speech and Language, vol. 14, no. 4, pp.373-400, 2000 Year. D. Hakkani-Tur, F. Bechet, G. Riccardi, G. Tur, “Beyond ASR 1-best: Using word confusion networks in spoken language understanding”, Computer Speech and Language, Vol. 20, No. 4, pp. 495-514, 2006.

However, in the above-described conventional technology (particularly, the technology described in Patent Document 1), it is possible to extract a keyword by recognizing the sound of a program or the like and obtain information related to the keyword from the outside. Cannot be searched.
Further, in the prior art, it is difficult for the user to understand the relationship between information acquired from the outside and the sound of the program itself.
In addition, the accuracy of speech recognition may be reduced due to changes in language expressions in broadcasting such as television and radio.

Further, in the lattice data processing method described in Non-Patent Document 1, clustering is performed based on the edit distance between word hypotheses obtained by converting edge clustering into phoneme strings, so that there is a problem that it takes time to compress the lattice. . That is, there is a problem that the compression procedure takes time because the similarity of pronunciation is compared for each word notation.
In addition, the lattice data processing method described in Non-Patent Document 2 can compress the lattice at a higher speed than the method described in Non-Patent Document 1, but does not perform word hypothesis clustering. There is a problem that the rate is low. That is, there is a problem that the accuracy of the compression lattice is not good because the similarity of pronunciation is not compared.

  The present invention has been made on the basis of the above problem recognition, and can store video and audio, and can search video and audio for the content of the utterance. It is an object of the present invention to provide a voice information extraction device that can present voice information such as information about a topic or a speaker to a user in an easy-to-understand manner.

  It is another object of the present invention to provide a speech information extraction apparatus having a configuration in which the accuracy of speech recognition does not decrease even if the language expression changes.

  Furthermore, another object of the present invention is to provide a speech information extraction device that can compress and use lattice data obtained as a result of speech recognition processing at high speed and a high compression rate.

[1] In order to solve the above-described problem, an audio information extraction device according to an aspect of the present invention includes a correspondence relationship between a video / audio storage unit that stores video and audio, a word, and an utterance time in audio. A search index storage unit for storing a search index, speech information, speech time, speech content that is a string of words, a topic, and voice information that associates at least one of a speaker name and speaker attribute with each other A speech information storage unit, an acoustic model storage unit that stores an acoustic model that statistically represents the acoustic features of speech, a language model storage unit that stores a language model that statistically represents the appearance frequency of words, A video / audio recording unit that acquires video and audio from outside and writes the video and audio to the video / audio storage unit, and the acoustic model read from the acoustic model storage unit and the language model storage unit. The speech recognition unit that performs speech recognition processing of the speech acquired by the video and audio recording unit using the language model that has been issued, and outputs a speech recognition result, and acoustic features for each speaker or each speaker attribute Corresponding to the voice acquired by the video / audio recording unit using the speaker data storage unit that preliminarily stores the speaker data statistically representing the voice data and the speaker data read from the speaker data storage unit A speaker identification unit that calculates and outputs a speaker name or speaker attribute, a text data acquisition unit that externally acquires text data related to the video and the audio acquired by the video / audio recording unit, and the text A topic extraction unit that extracts a topic by comparing the text data acquired by the data acquisition unit and the speech recognition result output by the speech recognition unit; and the speech recognition result; A speech information integration unit that writes speech information formed by integrating the topic and at least one of the speaker name or the speaker attributes into the speech information storage unit; and the search index based on the speech recognition result A search index creation unit that creates the data and writes it to the search index storage unit, and searches the search index storage unit and the voice information storage unit based on a search request based on a search term, and corresponds to the search term The audio information associated with the video and the audio is read from the audio information storage unit and presented to the search source as a search result, and the video and the audio stored in the video / audio storage unit are reproduced. A search server unit that enables the search server unit.
Here, video and audio are electrical signals or data representing video and audio, respectively. These can be processed by a computer or the like.
The utterance time is information represented by a combination of a program ID (identification information such as a unique numerical value determined from a broadcast channel and a program name) and an utterance start time. This utterance start time is represented by a relative time from the start of the program or an actual date and time (for example, Japan Standard Time).
The voice information is information about voice, and details thereof will be described later.

  According to the above configuration, the speech recognition result and the extracted topic are integrated as speech information and stored in the storage unit, and the search index is stored in the storage unit. On the other hand, a program having matching utterance content (speech recognition result) and a topic related to the utterance content can be presented to the user as a search result. In addition, the video and audio of the program selected from the search result can be reproduced and displayed.

[2] Further, according to one aspect of the present invention, in the speech information extraction device, the topic extraction unit includes a predetermined number of word sets included in the text data in a predetermined number of words of the speech recognition result. The topic is extracted from the text data by calculating the similarity based on the number of the images and by associating the text data with the speech recognition result based on the similarity.
With this configuration, the topic can be extracted and the topic boundary can be specified in the speech recognition result.

[3] Further, according to one aspect of the present invention, in the speech information extraction device described above, the appearance frequency of words in the text data acquired by the text data acquisition unit is calculated and stored in the language model storage unit. And a language model learning unit for updating the language model.
As a result, the language model can be updated based on the latest broadcast content, leading to an improvement in the recognition rate of speech recognition.

[4] Further, according to one aspect of the present invention, in the speech information extraction apparatus, the lattice data representing a directed acyclic graph of a word hypothesis output as the speech recognition result by the speech recognition unit is compressed. A lattice compression unit is further included, and the search index creation unit creates the search index based on the lattice data compressed by the lattice compression unit.
Thereby, the lattice of the speech recognition result can be compressed, and the storage capacity required for the speech recognition result lattice can be reduced.

  [5] According to another aspect of the present invention, in the above-described speech information extraction device, the search request is transmitted to the search server unit using a search term based on an input from a user, and the search server unit The search result is further displayed on the screen, and further, a search client unit that reproduces the video and the audio corresponding to the user's operation is further provided.

In addition, according to one aspect of the present invention, the lattice compression unit performs the following processes (1) to (3) in the audio information extraction apparatus.
(1) For the edges on the lattice, of the edges where the utterance start time and utterance end time overlap, the edges having the same notation are clustered (that is, the start and end of the edge are represented by those having a large posterior probability). , Give the sum of the posterior probabilities to the larger one (representative)).
(2) For overlapping edges, cluster overlapping edges (that is, have the same start and end nodes).
(3) Visit the nodes in the topological order for the edges on the lattice and merge the links.

  According to the present invention, it is possible to store video and audio, and to search video and audio by a search word for the content of the utterance, and information on topics and speakers at a corresponding portion of the audio as a search result Such voice information can be presented to the user in an easy-to-understand manner.

It is the block diagram which showed the function structure of the audio | voice information extraction device by embodiment of this invention. It is the schematic which shows the structure of the audio | voice information which the audio | voice information storage part 16 in the embodiment memorize | stores. It is the schematic which shows the structure of the screen displayed on the display apparatus provided in the search client part 20 in the embodiment. It is the schematic which shows the screen structure of the display of the search result in the display apparatus provided in the search client part 20 by the embodiment. It is a flowchart which shows the procedure of the language model learning process by the text collection part 3 and the language model learning part 9 in the embodiment. It is a flowchart which shows the procedure of the topic extraction process by the topic extraction part 14 in the embodiment. It is the 1st of the flowchart which shows the procedure of the process which compresses the lattice of the speech recognition result by the lattice expansion / compression part 12 in the embodiment. It is the 2nd of the flowchart which shows the procedure of the process which compresses the lattice of the speech recognition result by the lattice expansion / compression part 12 in the same embodiment. It is the 3rd of the flowchart which shows the procedure of the process which compresses the lattice of the speech recognition result by the lattice expansion / compression part 12 in the same embodiment. It is a flowchart which shows the procedure of the inverted index creation process by the search inverted index creation part 15 in the embodiment. It is the schematic which shows the data structure of the transposition index memorize | stored in the transposition index memory | storage part 17 for a search in the embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a functional configuration of the audio information extracting apparatus according to the embodiment. As shown in the figure, the audio information extraction device 50 includes a video / audio recording unit 1, a program information collection unit 2, a text collection unit 3, a speaker data storage unit 4, a speaker identification unit 5, and a voice recognition unit. Unit 6, acoustic model storage unit 7, language model storage unit 8, language model learning unit 9, text data storage unit 10, word dictionary storage unit 11, lattice expansion / compression unit 12 (lattice compression unit) , Audio information integration unit 13, topic extraction unit 14, search inverted index creation unit 15 (search index creation unit), audio information storage unit 16, search transposition index storage unit 17, and video / audio storage The unit 18 includes a search server unit 19 and a search client unit 20.

  The speaker data storage unit 4, the acoustic model storage unit 7, the language model storage unit 8, the text data storage unit 10, the word dictionary storage unit 11, the speech information storage unit 16, and the transposed index storage for search. The unit 17 and the video / audio storage unit 18 are realized using a magnetic disk device (HDD) or a semiconductor memory (such as a semiconductor RAM or a semiconductor ROM), respectively.

  The video / audio recording unit 1 receives video and audio input of broadcasting (general television, educational television, satellite broadcasting, radio first, radio second, etc.) and converts them into a digital video data file that can be read by a computer. Convert. The video / audio recording unit 1 writes the digital moving image data file obtained here into the video / audio storage unit 18. The obtained digital moving image data file is also used in a speaker identification unit 5 and a voice recognition unit 6 described later.

  The program information collection unit 2 acquires program information and EPG (Electronic Program Guide) information from an external server computer (such as a web server) via a communication line such as the Internet. These pieces of information include text information such as program titles and program performers. The program information collection unit 2 processes the acquired EPG information and the like, and stores it as metadata of the digital moving image data file converted by the video / audio recording unit 1.

  The text collection unit 3 acquires web text information from an external server computer (web server, etc.) via a communication line such as the Internet, and performs natural language processing such as morphological analysis on the web text information. Above, it writes in the text data storage unit 10. Here, the web text information acquired by the text collection unit 3 is, for example, web text information such as news posted on a website operated by a television or radio broadcasting station. The text data written in the text data storage unit 10 is read and used by the language model learning unit 9 and the topic extraction unit 14 as will be described in detail later.

  The speaker data storage unit 4 includes speaker data that statistically represents acoustic characteristics for each speaker or each speaker attribute, a word string or phrase at the end of the utterance, and a speaker or story immediately after the word string. The probabilities of changing person attributes are tabulated and stored in advance as language feature quantities. Here, the speaker attribute is an attribute associated with different acoustic features such as the gender (male or female) of the speaker.

  The speaker identification unit 5 uses the speaker data read from the speaker data storage unit 4 to analyze the voice acquired by the video / audio recording unit 1 and calculates a corresponding class (speaker name and speaker attribute). And output. Specifically, the speaker identification unit 5 obtains the acoustic feature amount based on the sound acquired by the video / audio recording unit 1, and the acoustic feature is stored in the speaker data storage unit 4. , The probability is determined as to whether it belongs to a speaker class (corresponding to individual speakers and speaker attributes) or unknown. Further, the speech recognition result is obtained from the speech recognition unit 6, and the table stored in the speaker data storage unit 4 is referred to from the last word string or phrase, and the speaker or speaker attribute is changed. Find the probability. The speaker identifying unit 5 integrates the acoustic feature value and the probability obtained from the language feature value, and the speech belongs to any speaker class stored in the speaker data storage unit 4 or is unknown Is determined. If the speaker class is known, the speaker identification unit 5 outputs the corresponding speaker name or speaker attribute as the identification result. If the speaker class is unknown, the speaker identification unit 5 generates a new speaker class. The number is output as an identification result.

  The voice recognition unit 6 performs voice recognition processing of the voice acquired by the video / audio recording unit using the acoustic model read from the acoustic model storage unit 7 and the language model read from the language model storage unit 8, and the voice recognition result Is output. At this time, in the voice recognition unit 6, as a preprocessing for voice recognition, first, the voice is a part of a human voice or a music part (a part not corresponding to a human voice). To identify. Then, the speech recognition unit 6 outputs metadata indicating that the section determined to be music is a music location together with the start time. For a section determined as speech, the section is recognized as speech, and the utterance content is output as a speech recognition result. The voice recognition processing itself by the voice recognition unit 6 uses existing technology. As will be described later in detail in the description of the processing of the lattice expansion / compression unit 12, the speech recognition unit 6 uses a word hypothesis as an edge and has a node corresponding to the time between words. Lattice structure data that is a directed acyclic graph is output as a speech recognition result. The lattice structure data is data representing hypotheses of speech recognition results and the probabilities of these hypotheses. It is possible to calculate the posterior probability and obtain the maximum likelihood word string by scanning the data of this lattice structure with a forward-backward algorithm or the like.

  The acoustic model storage unit 7 stores an acoustic model as data that statistically represents a relationship between a linguistic unit such as a phoneme and an acoustic feature amount when the phoneme is uttered as speech. . Specifically, the acoustic model is represented as a set of data in which phoneme unit notation, acoustic feature amount, and probability value are associated with each other. As the acoustic model, for example, a hidden Markov model (HMM) is used.

  The language model storage unit 8 stores a language model as data that statistically represents the frequency (feature) of appearance of linguistic units such as phonemes and words in a predetermined language. Specifically, as a language model, for example, a word n-gram is used. This word n-gram is a sequence of n consecutive words (n is a natural number) appearing in the text and an appearance probability (a real number of 0 or more and 1 or less) representing the appearance frequency of the n word strings. This is statistical data that accumulates tuples.

  The language model learning unit 9 reads the text data acquired by the text data acquisition unit 3 from the text data storage unit 10, and statistically calculates the appearance frequency of words in the text data, thereby storing the language model storage unit 9 in the language model storage unit 9. The process of updating the stored language model is performed. Details of the language model learning process will be described later with reference to a flowchart.

  The text data storage unit 10 stores the text data acquired by the text acquisition unit 3 from an external web server or the like via the Internet or the like. The text data is a news manuscript or the like that has been subjected to morphological analysis processing.

  The word dictionary storage unit 11 stores word dictionary data used in processing by the text acquisition unit 3 and the language model learning unit 9.

  The lattice expansion / compression unit 12 performs processing for compressing lattice data (a directed acyclic graph based on word hypotheses) as a speech recognition result output by the speech recognition unit 6. Note that the lattice expansion / compression unit 12 performs a process of expanding a lattice based on a bigram into a lattice based on a trigram and then compressing the lattice. Details of the processing by the lattice expansion / compression unit 12 will be described later in detail with reference to a flowchart.

  The voice information integration unit 13 integrates at least the voice recognition result (word string, text of utterance content) obtained from the voice recognition unit 6 and the topic obtained from the topic extraction unit 14, and the voice information storage unit as voice information. 16 is written. The voice information integration unit 13 also associates the identification result of the speaker or speaker attributes (for example, the gender of the speaker) output from the speaker identification unit 5 with the utterance content as part of the voice information. Write to the voice information storage unit 16. Furthermore, the audio information integration unit 13 generates a unique expression that indicates a specific thing such as a broadcast program theme, jingle, music such as sound effects, a person name, a place name, an organization name, or a structure composed of a plurality of words. Are also integrated as part of the voice information and written to the voice information storage unit 16. The data structure of audio information will be described later.

  The topic extraction unit 14 reads the text data acquired by the text data acquisition unit 3 from the text data storage unit 10, and extracts the topic by comparing the text data with the speech recognition result output from the speech recognition unit 6. Perform processing. More specifically, the topic extraction unit 14 calculates the degree of similarity based on the number of word groups (such as triplets) included in the text data included in the predetermined number of words in the speech recognition result. The topic is extracted from the text data by associating the text data with the voice recognition result based on the similarity, and the details of the processing of the topic extraction unit 14 will be described later. While explaining.

  The search transposition index creation unit 15 creates data of a search transposition index (search index) based on the speech recognition result by the speech recognition unit 6 and writes the data into the search index storage unit 17. In this embodiment, the search inverted index creation unit 15 creates a search inverted index based on the lattice data compressed by the lattice expansion / compression unit 12. The data structure of the search inverted index will be described later.

  The voice information storage unit 16 stores voice information. Here, the speech information is information including a program ID, speech start time, text of speech content (word string), speaker name, speaker sex, music (non-speech information), topic, and unique expression. This voice information is information obtained by processing of each part of the speaker identification unit 5, the voice recognition unit 6, and the topic extraction unit 14.

  The search transposed index storage unit 17 stores a search transposed index created based on the speech recognition result. This transpose index for search includes information on correspondence between words and utterance times in the voice. Here, the utterance time in this embodiment is specified by a combination of a program ID for identifying a program and an utterance start time.

  The video / audio storage unit 18 stores a digital moving image data file obtained by the video / audio recording unit 1. This digital moving image data file includes video data and audio data.

  The search server unit 19 is configured to be able to read data from the audio information storage unit 16, the search transposition index storage unit 17, and the video / audio storage unit 18, and a search from the search client unit 20 using these data. A search process is performed in response to the request, and the search result data is returned to the search client unit 20 as a response. The search result data is audio information (information read from the audio information storage unit 16) obtained as a result of the search or a digital moving image data file (information read from the video / audio storage unit 18). .

  The search client unit 20 transmits a search request to the search server unit 19 based on the input from the user, and displays search result data returned from the search server unit 19 as a response to the search server unit 19. Thereby, the user can search the voice information and browse the search result.

  FIG. 2 is a schematic diagram showing the structure of audio information stored in the audio information storage unit 16. As shown in the figure, the audio information is tabular data, and includes a program ID, an utterance start time, an utterance content (word string), a speaker name, a speaker gender (speaker attribute), and music ( Non-voice information), topic, and specific expression items. The music (non-speech information) is music such as the theme music of a broadcast program, jingle, and sound effects. The specific expression is an expression composed of a plurality of words, and indicates a specific thing such as a person name, a place name, an organization name, or a structure.

FIG. 3 is a schematic diagram illustrating a configuration of a screen displayed on a display device provided in the search client unit 20. The client search unit 20 presents audio information and digital video data file information (video and audio) obtained as a result of the search to the user on this screen.
As shown in the figure, this screen is mainly composed of three elements. The 1st is a window (code | symbol 113) for displaying the program list to which audio | voice information was provided. The second is a window (reference numeral 111) for displaying the video / audio of the program selected from the program list. And the 3rd is a window (code | symbol 112) which displays a speech recognition result (utterance content).

  First, a window 113 for the first program list is arranged on the left side of the screen shown in the figure, and (a) a thumbnail image display area (reference numeral 102) representing a typical scene of the program, (b) A program title display area (reference numeral 103) and (c) a list of topics included in the program (reference numeral 104) are combined into a set corresponding to one program and displayed for a plurality of programs. ing. These multiple programs are arranged vertically, so that newer programs are displayed above and older programs are displayed below. The program title displayed here is based on data originally acquired by the program information collection unit 2 and is included as metadata in the digital moving image data file. The search client unit 20 reads the program title from the metadata and displays it in the display area 103. The topic list displayed here is information originally extracted by the topic extraction unit 14. The search client unit 20 reads topic data from the audio information and displays the data in a list in the display area 104. The thumbnail image is a still image appropriately extracted from the digital moving image data file.

  Next, a second window 111 for displaying video / audio of the program displays video obtained by playing back the digital moving image data file. When the user performs an operation of clicking a thumbnail image displayed in the display area 102 or a program title displayed in the display area 103, the search client unit 20 selects the digital moving image data file of the program at the beginning of the program. Play from part. When the user performs an operation of clicking one of the topics displayed in the display area 104, the search client unit 20 selects a digital video data file of the program corresponding to the clicked topic ( Play from the starting point of the topic.

Various operation buttons and the like are displayed on the upper portion of the window 111. When the user operates these operation buttons and the like, the search client unit 20 starts or stops playback of the program. Or a process of changing the playback location.
Specifically, reference numeral 108 denotes a video / audio playback / stop button. When this button 108 is clicked when the video / audio is stopped, the search client unit 20 starts playback of the video / audio. If the button 108 is clicked when the video / audio is being played back, the search client unit 20 stops the playback of the video / audio.
Reference numeral 107 denotes a button for returning the playback position from the current playback position toward the start position by 30 seconds, and reference numeral 106 denotes a button for returning the playback position from the current playback position toward the start position by 10 minutes. Is a button for advancing the reproduction position from the current reproduction position toward the end position by 30 seconds, and reference numeral 110 is a button for advancing the reproduction position from the current reproduction position toward the end position by 10 minutes. When the user clicks one of these buttons 106 to 110, the search client unit 20 performs control to change the playback position of the video / audio according to each button.
Reference numeral 105 denotes a slider for moving the playback position to an arbitrary position from the start position to the end position. When the user performs an operation of moving the slider 105, the search client unit 20 Control is performed to change the playback position of the video / audio at a position corresponding to the position of the slider 105 where the slider 105 is moved.

  Next, a third window 112 is for displaying a speech recognition result (utterance content) corresponding to the program. The search client unit 20 displays the text of the utterance content on the window 112 and synchronizes with the video / audio being reproduced, and highlights the word of the utterance content corresponding to the currently reproduced position. As a highlighting method, for example, a method of displaying the background of the word in a color different from the normal background color (so-called highlight display) is used. That is, as the video / audio reproduction progresses, the highlighted words are sequentially shifted. This is realized by storing the utterance time for each word based on the voice recognition result by the voice recognition unit 6 and highlighting the currently uttered word along the elapsed time at the time of reproduction. In addition, when a speaker name or speaker attribute is obtained as voice information, the speaker name or speaker attribute may be displayed together.

  Further, the screen shown in FIG. 3 is provided with a text input unit 100 and a search button 101 for searching. When the user presses the search button 101 after inputting a search term to the text input unit 100 by operating a keyboard or the like, the search client unit 20 searches for the search term including the search term input to the search server unit 19. Send a request. The search server 19 performs a morphological analysis on the search word and searches the index using the search word that has been subjected to the morphological analysis. When the search result data is obtained from the response from the search server 19, the search client unit 20 displays the search result in the window 113 instead of the program list.

FIG. 4 is a schematic diagram showing the configuration of the search result display screen. As described above, this search result is displayed in the window 113. As shown in the figure, the window 113 when displaying the search result is (a) a display area (reference numeral 120) for displaying a representative image corresponding to the utterance including the search word used at the time of the search as a thumbnail image. ), (B) program title display area (reference numeral 121), (c) utterance start time display area that matches the search term in the program (reference numeral 122), and (d) the utterance content Display area (reference numeral 123).
The search client unit 20 displays a set of the display area 122 and the display area 123 as many as the number of appearances of utterances matching the search word in the program.
When a plurality of programs are matched as a search result, the search client unit 20 displays each of the programs.
In the display shown in the figure, the displayed program title is based on the data originally acquired by the program information collection unit 2 and is included as metadata in the digital moving image data file. . The displayed thumbnail image is a still image that is appropriately extracted from the digital moving image data file.

Next, detailed processing procedures of the text collecting unit 3 and the language model learning unit 9 will be described.
FIG. 5 is a flowchart showing a processing procedure by the text collection unit 3 and the language model learning unit 9.
In step S201, the text collection unit 3 performs a data source check at a predetermined time interval. In other words, the text collection unit 3 accesses an external computer such as a broadcast station website server, and compares it with the website data at the previous access, so that a new news manuscript or topic text is obtained from this time. Check if data is available. If new data is obtained (step S201: YES), the process proceeds to the next step S202. If not obtained (step S201: NO), the process returns to step S201, and further after the predetermined time has elapsed. Repeat the source check process.

  Next, in step S <b> 202, the text collection unit 3 performs a morphological analysis process on the text data obtained in step S <b> 201 and writes the result in the text data storage unit 10. Here, the morphological analysis process itself uses an existing technique. As a result of the processing in this step, text data (news manuscript or the like) divided into words is stored in the text data storage unit 10.

  In step S204, the language model learning unit 9 monitors the accumulation state of data in the text data storage unit 10 and checks whether or not new data is accumulated in a predetermined amount or more. If new data is accumulated in a predetermined amount or more (step S204: YES), the process proceeds to the next step S205. If not (step S204: NO), the process returns to step S201.

  Next, in step S205, the language model learning unit 9 reads new data from the text data storage unit 10 and performs a process of creating a language model based on the data. At this time, the language model learning unit 9 refers to the dictionary data read from the word dictionary storage unit 11. As described above, the language model created here is an n-gram, and the language model learning unit 9 performs each of n consecutive word strings based on the morphological-analyzed text data read from the text data storage unit 10. The language model data is created by counting the frequency of occurrence and statistical processing. Based on the result, the language model learning unit 9 rewrites the data in the language model storage unit 8.

  In step S206, the language model learning unit 9 notifies the voice recognition unit 6 to reload the updated data of the language model storage unit 8. Based on the notification, the speech recognition unit 6 reloads the language model, so that the speech recognition unit 6 can always perform speech recognition processing using the latest language model.

FIG. 6 is a flowchart illustrating a processing procedure performed by the topic extraction unit 14. Below, the detail of the process by the topic extraction part 14 is demonstrated.
In this process, the topic extraction unit 14 compares the first m words of text data such as a news manuscript obtained from the website with the m words from the start of utterance in the utterance content acquired from the speech recognition unit 6, It is determined which text data the speech recognition result matches by calculating the similarity between the two. Note that m is a positive integer.
In performing the processing by the topic extraction unit 14, the speech recognition unit 6 assigns a series of numbers starting from 1 to the speech recognition result in advance. Further, the latest K files (K is a positive integer) of the text collected from the website by the text collection unit 3 are processed by the topic extraction unit 14, and a series of K files starting from 1 is also set. The number is given.

Hereinafter, description will be made along the flowchart of FIG.
In step S <b> 301, the topic extraction unit 14 acquires a speech recognition result (utterance content) from the speech recognition unit 6. The speech recognition result acquired here is a maximum likelihood word string based on the posterior probability.
Next, in step S302, the topic extraction unit 14 sets (initializes) the variable n to 1.
In step S303, the topic extraction unit 14 extracts the first m words of the nth utterance.
In step S304, the topic extraction unit 14 starts with the first m words of the kth (k = 1, 2,..., K) text data read from the text data storage unit 10 and the m words extracted in step S303. The similarity between is calculated. The similarity between the first m words of the nth utterance and the first m words of the kth text data is defined as follows, for example. That is, the similarity is the number of triples (word pairs) included in m words of k-th text data included in m words of the nth utterance.
In step S305, the topic extraction unit 14 determines whether the calculated similarity is greater than or equal to a threshold value. This threshold value is appropriately determined and set in advance. If the similarity is equal to or higher than this threshold (step S305: YES), the process proceeds to step S307. If the similarity is less than this threshold (step S305: NO), the process proceeds to step S306.
In step S306, the topic extraction unit 14 increments the variable n (n ← n + 1). When the process of step S306 is completed, the process returns to step S303.
In step S307, the topic extraction unit 14 sets the nth utterance as the starting point of the kth topic. That is, the topic extraction unit 14 adds topic boundary information to the speech recognition result data. As a result, the speech recognition result can be divided at the topic boundary, and the topic can be stored in association with the divided result.
The topic extraction unit 14 performs the series of processes from steps S301 to S307 described above on each of the text data from the first to the Kth.

7, 8, and 9 are a series of flowcharts showing a processing procedure by the lattice expansion / compression unit 12. The lattice expansion / compression unit 12 expands and compresses the lattice of the speech recognition result by a method improved from the conventional method described in [Non-Patent Document 1] and [Non-Patent Document 2].
The speech recognition unit 6 outputs lattice structure (directed acyclic graph) data representing the speech recognition result. This data is a directed graph in which a word of the speech recognition result is an edge and any one of a start point, an intermediate point, and an end point is a node. There is one start point node and one end point node, and there are usually a plurality of intermediate point nodes. Each of these nodes corresponds to a predetermined time. That is, when there is an edge that starts at node A and ends at node B, the time at node A is the start time of the word corresponding to the edge, and the time at node B is the end time of the word corresponding to the edge It is. All nodes are connected and can reach all intermediate nodes by following the edge from the start node, and can reach the end node by following the edge from any intermediate node It is. The output from the speech recognition unit 6 is a hypothesis of a speech recognition result with a probability, and the paths that are parallel between the start point and the end point (that is, a plurality of paths that overlap in time) correspond to hypotheses that oppose each other. Is.
In the present embodiment, such a lattice structure is expressed by relational data having nodes and edges as entities, and exchanged between the processing units.
The lattice is based on a bigram language model that combines two adjacent words.

Hereinafter, it demonstrates along this flowchart.
First, in step S <b> 401 of FIG. 7, the lattice expansion / compression unit 12 acquires the above-described lattice structure speech recognition result data from the speech recognition unit 6.
Next, in step S402, the lattice expansion / compression unit 12 expands the lattice acquired above into a lattice based on a trigram language model that combines three consecutive words. This unfolding process itself uses the above-described conventional technique.

  Next, in step S403, the lattice expansion / compression unit 12 scans the lattice obtained above using a forward-backward algorithm, and calculates a posterior probability. Then, a route (maximum likelihood sequence) having the maximum posterior probability is acquired, and a graph p serving as a basis of the compression lattice is constructed.

Next, in steps S404 to S409, the lattice expansion / compression unit 12 performs edge clustering processing. The details of this clustering process are as follows.
That is, in step S404, the lattice expansion / compression unit 12 sorts the set elements from the edge set E {e ₁ , e ₂ , e ₃ , e ₄ ,. e ′ ₁ , e ′ ₂ , e ′ ₃ , e ′ ₄ ,..., e ′ _m ,.
In step S405, the lattice expansion / compression unit 12 initializes a variable n for clustering to 1.
The next steps S406 to S409 are processes for sequentially scanning the elements of the list.
When the lattice expansion / compression unit 12 extracts the n- th edge e ′ _n of the list, the cycle in which the overlap of the utterance times is larger than a predetermined threshold value and the word expressions on the edges are the same If there is a completed (scanned) m-th edge e ′ _m (n> m) (step S406: YES), in step S407, the edge e ′ _n is removed from the edge set E and the posterior of the edge e ′ _m . Add the posterior probability of edge e ' _{n to} the probability. If the determination result in step S407 is negative, step S407 is skipped and the process proceeds to the next step S408.
The lattice expansion / compression unit 12 then increments the variable n for clustering (n ← n + 1) in the next step S408.
In step S409, the lattice expansion / compression unit 12 determines whether or not all edge clustering has been completed. If all the processing has not been completed (step S409: NO), the process returns to step S406 in order to perform the same processing for the remaining edge set. If all have been completed (step S409: YES), the process proceeds to the next step S410.

Next, in steps S410 to S417 in FIG. 8, the lattice expansion / compression unit 12 performs edge aggregation.
First, in step S410, an edge list is generated by rearranging the elements of the edge set E in descending order of posterior probabilities.
In step S411, the lattice expansion / compression unit 12 initializes a variable n for aggregation to 1.

In step S412, the lattice expansion / compression unit 12 determines whether or not the posterior probability of the n-th edge e ′ _{n in} the edge list is equal to or greater than a predetermined threshold. _If the posterior probability of the edge e ′ _n is less than the predetermined threshold value (step S412: NO), the process proceeds to step S413. If the posterior probability is equal to or greater than the threshold value (step S412: YES), the process proceeds to step S414.
When the process proceeds to step S413, the lattice expansion / compression unit 12 removes e ′ _n from the edge set E, and the process proceeds to step S416.
When the process proceeds to step S414, in S414, the lattice expansion / compression unit 12 searches for an edge e ′ _m (where n> m) where the overlap of the utterance time is _equal to or greater than a predetermined threshold with respect to the edge e ′ n.
If such e ′ _m exists (step S 414: YES), in the next step S 415, the lattice expansion / compression unit 12 changes the start / end node of the edge e ′ _{m to} the start / end node of the edge e ′ _n. To do.
If the determination result in step S414 is negative, the process of step S415 is skipped and the process proceeds to the next S416.
In step S416, the variable n is incremented (n ← n + 1).
In step S417, the lattice expansion / compression unit 12 determines whether all the aggregation processing has been completed. If the processing in step S415 above is completed for all edges in the ascending order in the edge set (step S417: YES), the process proceeds to the next step S418, and if there are still remaining edges (step S417: If NO, the process returns to step S412 to process the next edge.

In steps S418 to S429 in FIG. 9, the lattice expansion / compression unit 12 obtains a compressed lattice by merging the edge set obtained by the clustering and aggregation into the sequence p.
First, in step S418, the lattice expansion / compression unit 12 obtains a list in which lattice node sets are arranged in a topological order.
In step S419, a variable k for merging is initialized to 1.
In step S420, the lattice expansion / compression unit 12 generates a list of edges starting from the node v _k from the edge set E.
In step S421, a variable l (L) is initialized to 1.
In step S422, the lattice expansion / compression unit 12 searches for the edge f _{h of the} compression lattice that maximizes the overlap of the utterance times for the _l- th edge e _l of the edge list.
In step S423, it is determined whether or not this f _h has been visited. If it has been visited (step S423: YES), the process proceeds to the next step S424, and if it has not been visited (step S423: NO), the process proceeds to step S425.
When step S424, the lattice expansion and compression unit 12 divides the terminal node of the f _h into two, creating a new node f on compressed lattice, copies the word notation and the posterior probability of the edge e _l . Then, the process proceeds to step S426.
When step S425, generates a new edge f connecting the start-end of a f _h, copies the word notation and the posterior probability of the edge e _l. At this time, it is assumed that f _h has been visited. Then, the process proceeds to step S426.
In step S426, the variable l is incremented (l ← l + 1).
In step S427, an edge list end determination is performed. If it has been completed (step S427: YES), the process proceeds to step S428. If it has not been completed (step S427: NO), the process returns to step S422.
Also, another end determination is performed in step S428, and if it has ended (step S428: YES), the process of the entire flowchart is ended, and if it has not ended (step S428: NO), the process proceeds to step S429. move on.
In step S429, the variable k is incremented (k ← k + 1), and the process returns to step S420.
That is, the lattice expansion / compression unit 12 performs the operations from step S422 to S425 for all the edges of the edge set E to obtain a compression lattice.

In short, the points of the above-described processing by the lattice expansion / compression unit 12 are as follows (1) to (3).
(1) For the edges on the lattice, of the edges where the utterance start time and utterance end time overlap, the edges having the same notation are clustered (that is, the start and end of the edge are represented by those having a large posterior probability). , Give the sum of the posterior probabilities to the larger one (representative)).
(2) For overlapping edges, cluster overlapping edges (that is, have the same start and end nodes).
(3) Visit the nodes in the topological order for the edges on the lattice and merge the links.
Thereby, the lattice data of the speech recognition result can be compressed at a higher speed and with a higher compression rate than the method according to the prior art.

Through the processing of the procedure described above, the lattice expansion / compression unit 12 compresses the expanded lattice based on the expanded lattice and creates a compressed lattice (confusion network).
When the total of posterior probabilities of edges connecting adjacent nodes exceeds 1 with respect to the compression lattice obtained in this way, a process of dividing the posterior probability of each of those edges by the sum of the posterior probabilities. Do. Conversely, if the sum of the posterior probabilities of the edges connecting adjacent nodes is less than 1, a new edge having an empty word notation is generated between those nodes, and the sum of the posterior probabilities of the edges is 1. The posterior probability value of the newly generated edge is set so that Here, the newly generated posterior probability value of the edge is 1− (the sum of the posterior probabilities of other edges).

  The amount of lattice data of the speech recognition result is enormous, but as described above, the lattice expansion / compression unit 12 compresses the lattice, so that the data can be easily handled and the processing speed can be increased. I can plan.

FIG. 10 is a flowchart showing a processing procedure performed by the search inverted index creation unit 15. A method for creating a search index will be described below with reference to this flowchart.
First, in step S <b> 501, the search inverted index creation unit 15 acquires lattice data for one utterance that is compactly compressed from the lattice expansion / compression unit 12. In the following steps, processing for each edge included in this lattice is performed.
Next, in step S502, the search inverted index creation unit 15 determines whether a word notation is assigned to the current edge. If it is assigned (step S502: YES), the process proceeds to the next step S503, and if it is not assigned (step S502: NO), the process jumps to step S504.
In step S <b> 503, the search inverted index creation unit 15 updates the search inverted index storage unit 17 by adding one record based on the word notation assigned to the current edge.

  FIG. 11 is a schematic diagram showing the data configuration of the inverted index stored in the search inverted index storage unit 17. As shown in the figure, this transposed index is tabular data, and has items of a word notation ID, a program ID, and an utterance start time. The word notation ID is data for uniquely identifying the word notation, and when the ID of the word assigned to the edge is not assigned, a new ID is assigned as an unsigned 32-bit integer. The program ID is data for uniquely identifying the broadcast program that is the target of voice recognition. The utterance start time is time information indicating the start position of the utterance in the program, with one utterance as a unit. This time information may be expressed as a relative time from the start of the program, or as an actual date and time (for example, Japan Standard Time). Since the inverted index has such a structure, the number of appearances of word notation for each program can be easily extracted from the search inverted index storage unit 17. That is, the search server unit 19 can utilize the information read from the search inverted index storage unit 17 when performing the above-described search process.

  Returning to FIG. 10, next, in step S <b> 504, the search inverted index creation unit 15 determines whether or not processing of all edges has been completed in the given lattice for one utterance. When all the edges have been processed (step S504: YES), the processing of the entire flowchart is terminated, and when there are still remaining edges (step S504: NO), the next edge is processed. Therefore, the process returns to step S502.

Note that the function of the above-described voice information extraction device is realized by an electronic circuit.
In particular, it is preferable that the function of the apparatus is realized by a single or a plurality of stored program type computers. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It is also possible to include those that hold a program for a certain time, such as a volatile memory inside a computer system serving as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

<Example>
In order to verify the operation of the above embodiment, a system was actually constructed. The outline is described below.
The video / audio recording unit 1 is configured to be able to acquire video and audio signals from the TV tuner, and actually receives and captures broadcasts of channels set at a preset date and time. The date and time and channel are automatically set based on the broadcast program guide and the EPG superimposed on the broadcast wave on the NHK (Japan Broadcasting Corporation) website. On the other hand, it can be done from the screen through the user interface. In addition, it is now possible to start / end the acquisition of video and audio and change the channel setting by button operation from the user.
The text collection unit 3 can obtain text information about broadcast programs from the NHK website via the Internet.
Each function constituting the voice information extracting apparatus 50 is realized by writing a program for a computer and executing the program on a plurality of computers linked via a LAN.
In the search client unit 20, the search result is displayed by the above-described method, and the video and audio selected by the user are reproduced and displayed.
Also, each acoustic feature of the announcer appearing in the program is stored in the speaker data storage unit 4 in advance, so that speaker identification is performed with high accuracy, and the speaker name is added together with the text of the speech recognition result. It was possible to display.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

By utilizing the present invention, it is possible to index broadcast programs and video / audio resources so that they can be easily searched and viewed.
Further, by using the present invention, it is possible to provide the extracted audio information to the metadata production system and perform a broadcasting service.
Further, by utilizing the present invention, it is possible to efficiently collect speech data and related text data in order to construct a statistical acoustic model and a statistical language model used in the speech recognition apparatus.

DESCRIPTION OF SYMBOLS 1 Video | video recording part 2 Program information collection part 3 Text collection part 4 Speaker data storage part 5 Speaker identification part 6 Speech recognition part 7 Acoustic model storage part 8 Language model storage part 9 Language model learning part 10 Text data storage part 11 Word Dictionary Storage Unit 12 Lattice Expansion / Compression Unit (Lattice Compression Unit)
13 voice information integration unit 14 topic extraction unit 15 transposed index creation unit (search index creation unit)
16 Voice information storage unit 17 Inverted index storage unit for search (index storage unit for search)
18 Video / Audio Storage Unit 19 Search Server Unit 20 Search Client Unit 50 Audio Information Extraction Device

Claims (5)

A video / audio storage unit for storing video and audio;
A search index storage unit that stores a search index including correspondence between words and speech utterance times, speech time, utterance contents that are word strings, topics, speaker names, or speaker attributes A voice information storage unit that stores voice information associated with at least one of the following: an acoustic model storage unit that stores an acoustic model that statistically represents the acoustic features of the voice;
A language model storage unit for storing a language model that statistically represents the appearance frequency of words;
A speaker data storage unit that pre-stores speaker data that statistically represents acoustic features for each speaker or speaker attribute;
A video and audio recording unit that acquires video and audio from outside and writes the video and audio in the video and audio storage unit;
Using the acoustic model read from the acoustic model storage unit and the language model read from the language model storage unit, the voice recognition processing of the voice acquired by the video / audio recording unit is performed, and a voice recognition result is output. A voice recognition unit that
Using the speaker data read from the speaker data storage unit, a speaker identification unit that calculates and outputs a speaker name or speaker attribute corresponding to the sound acquired by the video and audio recording unit;
A text data acquisition unit for acquiring text data related to the video and the audio acquired by the video / audio recording unit;
A topic extraction unit that extracts a topic by comparing the text data acquired by the text data acquisition unit and the speech recognition result output by the speech recognition unit;
A voice information integration unit that writes voice information obtained by integrating the voice recognition result, the topic, and at least one of the speaker name and the speaker attribute into the voice information storage unit;
A search index creation unit that creates data for the search index based on the speech recognition result and writes the data in the search index storage unit;
The search index storage unit and the audio information storage unit are searched based on a search request based on a search word, and the audio information associated with the video and the audio corresponding to the search word is read from the audio information storage unit. A search server unit that presents the search source as a search result and that can reproduce the video and the audio stored in the video and audio storage unit;
An audio information extraction apparatus comprising: The topic extraction unit calculates the similarity based on the number of word sets included in the text data included in the predetermined number of words of the speech recognition result, and based on the similarity, the text data and the text data Extracting the topic from the text data by performing a correlation with a speech recognition result;
The audio information extracting apparatus according to claim 1, wherein   It further comprises a language model learning unit that updates the language model stored in the language model storage unit by calculating the appearance frequency of words in the text data acquired by the text data acquisition unit. The voice information extracting device according to claim 1 or 2. A lattice compression unit for performing a process of compressing lattice data representing a directed acyclic graph of the word hypothesis output as the speech recognition result by the speech recognition unit;
The search index creation unit creates the search index based on the lattice data compressed by the lattice compression unit.
The speech information extraction device according to any one of claims 1 to 3, wherein   The search request is transmitted to the search server unit using a search term based on an input from the user, the search result from the search server unit is displayed on the screen, and further based on an operation from the user The audio information extracting apparatus according to claim 1, further comprising a search client unit that reproduces the corresponding video and audio.

Images