JPH0749695A - Time sequential data recording and reproducing device - Google Patents

Abstract

PURPOSE:To reproduce meaningful information only by performing recognition processes of time sequential data, retrieving key data, recording the constructional information generated by the above processes with the time sequential data and retrieving the data based on the key data. CONSTITUTION:Time sequential data are inputted through a time sequential data inutting means 1, stored in a storage means 2 and sent to a construction analysis means 3. Recognition processes of the time sequential data are performed in the means 3, key data are detected, a constructional information is generated and is stored in a constructional information storage means 4. A user inputs key data into a retrieving instruction input means 5 and the instruction is sent to a retrieving means 6. The means 6 retrieves all constructional information corresponding to the key data from the means 4, the time sequential data are retrieved from the means 2 and transmitted to an information output means 7. Moreover, a key data input means 8 accepts corrections and additions so that not only corrections but also changes in the constructional information are made by the user on the key data detected by the means 3.

Description

Detailed Description of the Invention

[0001]

The present invention stores time series data,
The present invention relates to a time series data recording / reproducing device that reproduces stored time series data.

[0002]

2. Description of the Related Art In recent years, in order to store and reproduce multimedia data such as voices and images and to perform editing processing against the backdrop of improvement in processing capacity of computers, recording and retrieval of multimedia data are performed in various forms. Is possible.

Large-capacity recording media have become relatively easy to use, and although applications for multimedia data input / output, processing, and editing processing have been developed, these have only an editing function by putting multimedia data in and out. It has only been realized.

The recording of a huge amount of multimedia data can be realized by using a recording medium having a sufficient recording capacity. However, in the search and output of recorded multimedia data, the greater the amount of data, the greater the labor of the computer for finding the necessary part from the stored data.
It cannot be said that it has been effectively realized.

Particularly, when the data to be handled is time-series data, the time required for the search becomes long in proportion to the time length of the data. In addition, the information that the user wants to search and output is not necessarily the huge amount of multimedia data itself, but rather a meaningful part of the multimedia data, or the meaning itself, that is, the key. It is often data.

This key data cannot be obtained by the conventional simple recording / reproducing / editing processing of multimedia data. For example, if the voice data “Um, I want one, no, three orange juices” is input, rather than outputting the input voice data as is, “I want three orange juices.”
It is easier for the user to output the information that has the meaning of the meaning.

Therefore, multimedia recognition processing is performed to detect key data, and at least the detected key data and structural information composed of link information for associating the key data with the time-series data are created, and the key information is generated. It is necessary to output some combinations of structural information including data as meaningful parts.

[0008] The first problem in the structure analysis process for automatically generating structure information such as the recognition process is the problem of real-time process and error of detected key data. The problem of real-time processing is that it is becoming possible to sufficiently realize the processing of recognizing voices, images, etc. in real time by improving the calculation function.

The problem of the error of the structural analysis result is that the user may correct the key data automatically generated by the key data generation process, assuming that the key data may include an error in advance. By performing update processing such as the user directly adding the key data itself,
More accurate key data can be created. However, there is no multimedia data recording / retrieving device that uses the key data as a retrieval key for later retrieval.

[0010]

In the conventional multimedia time-series data recording / retrieving apparatus as described above, it is possible to realize an editing function such as recording / playback / editing processing by simply inserting / removing the multimedia time-series data. there were.

Further, although recognition processing of multimedia time-series data is being realized in real time, key data meaningful to the user cannot be obtained.

According to the present invention, it is possible to take out only a meaningful portion required for a user from multimedia time-series data and reproduce it, or to output meaningful information in the time-series data. It is an object to provide a series data recording / reproducing device.

Further, it is necessary to improve the system in order to improve the structural analysis processing capacity for generating structural information, but it is an object of the present invention to provide a system that improves based on the result of structural analysis processing. .

[0014]

In order to solve the above problems, the present invention is applied to a time series data recording / reproducing apparatus which stores time series data of multimedia and reproduces the stored time series data. The key information is detected by performing the recognition process of the time series data, and at least the detected key data, time information of the key data, and structural information composed of information for linking the time series data and the key data with each other. Structure information analyzing means for creating, structure information storing means for storing the structure information, and searching means for searching the time series data from information linking the time series data of the structure information with the key data as a search key. It is characterized by including.

[0015]

According to the time series data recording / reproducing apparatus of the present invention,
When searching the time-series data or the information required by the user from the time-series data or the structural information, the time-series data or the information required by the user can be extracted using the key data as a search key.

Further, since the user can directly input the key data, even if the structure information automatically generated by the structure analysis means is erroneous or insufficient, the structure information recorded in the structure information storage means can be used. The user can modify / add the key data as a search key at any time while displaying the information on the screen and confirming it.

Further, even when the structure information is insufficient at the present time, necessary key data can be obtained by newly creating the structure information by utilizing the time series data already recorded in the time series data storage means. It is possible to reconstruct the structural information including the.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First, an embodiment of the present invention applied to a system for recognizing and processing time-series data such as voice and moving images (referred to as a recognition and understanding system) will be described. Here, not only recognition / understanding, but also a response system or a dialogue system that receives a result and outputs / responds in some way may be used.

For example, the key data is a result of processing such as recognition / understanding. The link information is information indicating the correspondence with the time series data from which the processing result is obtained. That is,
This is information indicating from which part of the time-series data the key data is obtained.

In general, processing such as recognition and understanding goes through a plurality of processing steps. For example, in the case of voice recognition and understanding, a plurality of processes such as voice segment detection, voice analysis, pattern recognition, and language processing are performed. Speech recognition based on word spotting (Tsuboi, Hashimoto, Takebayashi: "Continuous Speech Understanding Based on Keyboard Spotting" Technical Report of IEICE, SP-91-95, pp.33-40 (1991.12)) , Word detection,
It goes through processing such as syntax analysis, word analysis, and semantic analysis.

Thus, when a plurality of processing results are passed,
The key data may be not only the final processing result but also an intermediate processing result at each processing stage in the middle.

In this case, the link information may be information indicating the correspondence with the original input time series data, or information indicating the correspondence with the processing result in the previous processing stage.
For example, in the case of speech understanding based on the word spotting described above, detected words, a syntax tree composed of word strings, and the meaning obtained as an analysis result are used as key data, and the link information is the original speech. Information indicating correspondence with data (for example, time, sample point, or data address) may be used, or a word string or a syntax tree as an intermediate result may be indirectly used as the link information to correspond to the input voice data. Very good.

When a plurality of types of data, such as multimedia data, are handled as input time series data, the link information includes a data identifier indicating which time series data the key data corresponds to, The correspondence becomes clearer.

In the search, when the above-mentioned key data that can be output by the recognition / understanding system is specified, the structural information having the key data that matches the key data is searched, and the link information described in the structural information is also searched. It is realized by accessing the time series data of and.

An embodiment of the present invention will be specifically described with reference to the drawings. A procedure for inputting and storing time series data will be briefly described with reference to a block diagram of an embodiment of the time series data recording / reproducing apparatus of the present invention shown in FIG. 1 and a flow chart shown in FIG.

The time-series data is input from the time-series data input means 1 such as a microphone or a camera as shown in step "time-series data input" 200. The time-series data input means 1 sends the input time-series data to the time-series data storage means 2 as in step “Send time-series data to time-series data storage means” 201.

The time-series data storage means 2 executes the step "store time-series data in the time-series data storage means" 201.
Store the sent time-series data as in. Similarly, the time-series data input means 1 sends the input time-series data to the structure analysis means 3 as in the step “send time-series data to structure information analysis means” 203.

The structure analyzing means 3 detects the key data by performing the recognition process of the transmitted time-series data as in step "generate structure information" 204, and at least the detected key data and the key. Data time information,
Structural information composed of information that links the time series data and the key data is generated.

The structure information of the time-series data generated by the structure analysis means 3 is stored in the structure information storage means 4 as in the step "store structure information in structure information storage means" 205.
Stored in. The key data referred to here is information meaningful to the user that can take a part of the time-series data such as the result of pattern recognition processing such as voice recognition extracted from the input time-series data. is there.

The procedure of outputting the meaningful information of the time series data stored in the time series data storage means 2 and the time series data stored in the structure information storage means 4 according to the user's request is also shown in the flowchart of FIG. And briefly explained.

The user uses the step "input search command" 30
As shown in 0, the key data to be input as the search key is input to the search command input means 5. The search command means 5 sends a search command to the search means 6 as in the step "send search command to search means" 301.

The search means 6 uses the step "search" 3
02, the structure information storage unit 4 is searched for all the structural information corresponding to the key data in accordance with the search command input to the search command unit 5, and based on the information linked to the searched structural information. The time series data is retrieved from the time series data storage means 2.

The retrieval means 6 retrieves meaningful information such as a combination of keywords from the structure information storage means 4 based on the similarly retrieved structure information. The search means 6 carries out the step “Carry the searched data to the information output means” 303.
As described in 1., the retrieved time series data and meaningful information are sent to the information output means 7.

The information output means 7 visually or audibly presents the retrieved time-series data and meaningful information to the user as in step "output retrieved data" 304.

When the time-series data is presented to the user, the information output means 7 can output the time-series data and a meaningful part separately from other meaningful parts. The key data input means 8 is an input device such as a pointing device such as a mouse, a pen, a touch panel, or a keyboard.

The key data input means 8 receives correction and addition of key data from the user. When using these input devices, the information output means 7 displays the key data and the structural information on the screen and changes the structural information corresponding to the key data if there is an input from the user.

In this way, not only the key data detected by the structure analysis means 3 can be corrected, but also the user can change / add the structure information. Such input of key data by the user is performed while confirming the contents recorded in the structure information storage means 4 and the time-series data storage means 2 by the information output means 7, and a structure that meets the user's request. Information can be improved and the quality of structural information can be improved.

The case of inputting and outputting audio data as time series data will be specifically described below. The voice data is input by the time-series data input means 1 having an input device such as a microphone.

As shown in FIG. 4A, the time-series data input means 1 adds identification data such as data type and time information to the input voice data. The data type is information for reproducing input data such as data discretization information such as an input device, a channel number, a type of medium, and a sampling frequency. This information is hereinafter referred to as additional time series data.

A plurality of each input device can be used at the same time, and for example, multi-channel input using a plurality of microphones is also possible. In this case, as one piece of data type information, a channel number indicating from which channel the data was input is included.

The input start time indicates the time when the voice data is input to the system, but if the originally input multimedia data includes time information, it can be used as it is. It is possible.

FIG. 4B shows a time series data input means 1
However, in the voice data, "Data is input with a microphone, the channel of the microphone is 2, the data medium is voice, the sampling frequency is 12 kHz, the quantization accuracy is 16 bits, and
The information that the start time of the voice is ts and the end time is te is shown in the form of a table of additional time series data.

The time-series data input means 1 sends the additional time-series data of FIG. 4 to which the identification data is added to the time-series data storage means 2 and the structure analysis means 3. Time series data storage means 2
Is the additional time series data sent and the ID of the time series data
And information indicating the storage position of the additional time series data in which the addresses of the time series data storage means 2 are associated with each other. Figure 5
(A) shows that the time-series data with the time-series data ID “SP-129” is stored in the storage area of the address “xxxxx” of the time-series data storage means 2. FIG. 5B is a diagram in the form of a table in which the additional time series data of the time series data ID "SP-129" is stored in the address "xxxxxx".

The structure analysis means 3 detects the key data by recognizing the additional time series data, and links at least the detected key data, time information of the key data, the time series data and the key data. Structure information composed of information to be created is created. The structure information is configured to always include time information indicating from which part of the audio data, in addition to the structure information name or structure information ID indicating the type of structure information. This time information can be obtained based on the start time of the additional time series data and the discretization information of the data such as the sampling frequency. This structural information will be displayed as shown in FIG.

In the structure information of FIG. 6, the structure information ID is "WD-
5 ”, the original time-series data ID“ SP-129 ”, the structure information name is“ word detection result ”, the time information is the start time“ t1 ”, the end time“ t2 ”, and the detected key data. It indicates that it is composed of a certain keyword. The keyword detection by the recognition processing will be described in detail later.

The structure analysis means 3 sends the generated structure information to the structure information storage means 4, and the structure information storage means 4 associates the sent structure information with the ID of the structure information and the address in the structure information storage means. Information indicating the storage location of the attached structure information is stored. FIG. 7A shows that the structure information having the structure information ID “WD-5” is stored in the storage area of the structure information storage unit 4 at the address “◯◯◯◯”. FIG. 7 (b) is shown in the form of a table in which the structure information of the structure information ID "WD-5" is stored in the address "○○○○".

The speech data recognition processing by the structure analysis means 3 will be specifically described below. Here, as structural information,
An example will be described of detection of key data such as a voice section, a spoken word, utterance meaning contents, and utterance environment information obtained through a voice analysis process, a word detection process, a syntactic and meaning analysis process, a dialogue structure understanding process, and the like.

Here, the structure analysis means 3 is assumed to be composed of a voice section detection section 81, an acoustic analysis section 82, a word detection section 83, and a syntactic and meaning analysis section 84, as shown in FIG.
The voice section detection unit 81 detects a section in which voice exists in the voice data as key data. This is to check the power of the audio signal, and set the start point when the value exceeds a certain threshold for a certain period of time as the beginning of the voice section, and when the power value does not exceed the certain threshold for a certain period of time, the start point is the voice section. It is realized by setting the end of. Figure 9 for clarity
A graph showing the relationship between the time change and the voice power is shown in Fig. 9. The start time t1 (start end) and end time t2 of the voice in Fig. 9 are shown.
(End) will be detected. In this, the start point when the power value of the voice signal exceeds the threshold value for a fixed time (dt1) is the start of the voice section, and the power value is a fixed time (dt2).
When the threshold value is not exceeded at, it can be realized by setting the start point as the end of the voice section. For the threshold value of power and the length of duration, if different values are set for the start end detection and the end detection,
The detection accuracy can be improved. The structure information regarding the detection of the voice section is represented here as shown in FIG. 10, and the structure information ID “VP-013” and the ID of the original time series data are shown.
"SP-129", structure information name "voice section", start time "t1" as time information, end time "t2", and voice section "t1 to t2" as key data. Here, the key data is the same as the time information because the key data represents the voice section by time, but the key information and the time information must be clearly distinguished in the structure information.

The acoustic analysis unit 82 performs acoustic analysis of voice data. Here, spectrum analysis by a method such as FFT (Fast Fourier Transform), smoothing in the frequency domain, and logarithmic transformation are performed, and, for example, an acoustic analysis result, that is, a frequency spectrum pattern is obtained at 8 ms intervals from a 16-channel bandpass filter. To get

This acoustic analysis result, that is, the frequency spectrum pattern is sent to the word detecting section 83, and the word detecting section 83 is sent.
The word detection process is performed. The word detection unit 83 uses, for example, documents (Kanazawa, Tsuboi, Takebayashi “Word Detection from Continuous Speech Containing Unwanted Words” IEICE Technical Report, SP91-22,
pp.33-39 (1991.6)) can be used to perform word detection processing. As a result, the pattern of the frequency spectrum sequence of the input voice is compared with the standard pattern of the frequency spectrum sequence of the word to be detected, and a word with a high score can be obtained as word detection result information.

The word detection result information obtained by the word detection unit 83 is treated as a combination of the key data such as the start and end of the word and the similarity with the standard pattern, and the time information, the word name, and the word information are registered. It is possible to combine key data such as the likelihood of the word being detected with the detected word as structural information. As shown in FIG. 11A, here, the structure information for word detection is structure information ID “WD-7”, original time-series data ID “SP-129”, structure information name “word detection result”. , Start time “t1”, end time “t2”, word name (keyword) “yes”, likelihood “0.82” as time information
Composed of.

The words to be detected can be determined in advance by the user and can be added or deleted as necessary. Although the example of the processing of the word detection unit 83 described above has been described as having a standard pattern of frequency spectrum series for each detected word, it is of course possible to have a standard pattern of frequency spectrum series for each phoneme. It is also possible to use word-based matching and phoneme-based matching together.

By the way, the word detection result information detected by the word detection unit 83 does not necessarily directly become the final result of word recognition. Among the keywords obtained as word detection result information, in addition to the correct words to be detected that were included in the utterance, the frequency spectrum sequences that are not actually included in the utterance are similar, but It may also include mis-detected words. For example, the word “yes” and the word “large” (when read as “dai”) are similar to each other, and thus may be erroneously detected. here,
When “large” is detected as key data and structural information is generated, the structural information is structural information ID “WD-8” as shown in FIG. 11B, and the original time-series data ID “SP- 12
9 ”, structural information name“ word detection result ”, time information as start time“ t1 ”, end time“ t2 ”, word name (keyword)
It is composed of "Large" and likelihood "0.75".

The syntactic and semantic analysis unit 84 performs syntactic and semantic analysis on all possible word sequences (combinations of words that do not overlap in terms of time) with respect to the words obtained as a result of detection by the word detection unit 83, and accepts them. Semantic content of utterances generated by various word sequences are defined as semantic expression candidates.

The semantic expression candidates obtained here are not necessarily limited to one, and there may be a plurality of acceptable word series candidates. The meaning of the utterance is finally determined by selecting an appropriate meaning expression from the plurality of candidates in consideration of the history of the meaning content of the utterance up to that point. Therefore, the final result of word recognition refers to the word sequences that make up the semantic expression selected here. The process of determining the meaning of these utterances is described in the literature (Tsuboi, Hashimoto,
Takebayashi “Continuous Speech Understanding Based on Keyword Spotting” IEICE Technical Report, SP91-95, pp.33-40
(1991.12)) or the literature (Sadamoto, Shinchi, Tsuboi, Takebayashi "Dialogue processing of the unspecified speaker spoken dialogue system TOSBURG", Acoustical Society of Japan, 1-P-17, pp.137-138 (19)
It can be carried out by the method disclosed in 92.3)).

These are methods for obtaining a semantic expression from an expected sequence of words, which can be determined in advance from an expected sequence of words and their semantic representation. Also, the user can add or delete as necessary. The addition / deletion will be described in detail later.

The above-described meaning expression candidates and the word sequences forming the meaning expression candidates can be treated as key data relating to the meaning content candidates of the utterance. The structure information created from this key data is shown in FIG. 12. The structure information in FIG. 12 includes structure information ID “SR-5”, original time-series data ID “SP-129”, structure information. Name "utterance meaning", start time "t1", end time "t2", meaning "affirmation" of the utterance, the total number of constituent words is two, and the words "yes" and "yes" were selected as the meaning expression. Indicates whether or not. Further, the information about the word series can be linked to the structure information about the word detection result by having a pointer that points to the structure information obtained as the word detection result. Similarly,
The finally selected semantic expression and the word sequences that compose it can also be treated as structural information regarding the semantic content of the utterance.

The environment information extraction unit 85 extracts key data relating to the surrounding environment when the voice data is taken in, and creates structure information. By adding the environment information extraction unit 85, more detailed structure information can be obtained.

For example, a case where voices uttered by a plurality of speakers are treated as input data will be described with reference to FIG. 13A shows the voice power of the voice input from the microphone of the speaker A, and FIG. 13B shows the voice power input from the microphone of the speaker B in a graph. .

Further, even if a microphone is not prepared for each attendee, a large gain can be obtained in a specific direction by constructing a microphone array (a directional microphone), and a voice from the direction in which the speaker is present can be obtained. Can be taken out by strengthening, and the speaker can be specified. Therefore, by using these methods, key data indicating which speaker is speaking can be extracted and used as structural information.

By comparing the voice intervals of all speakers, the time when no speaker is speaking can be detected as silence. On the other hand, by using these methods, conversely, it is possible to extract the component of the ambient environment sound other than that of the speaker. That is, the portion of the voice data detected by the voice section detection unit 81 other than the voice section can be treated as the ambient sound. A graph showing the ambient sound input from the microphone of the speaker A is shown in FIG.
A graph showing the ambient sound input from the microphone of the speaker B is shown in FIG. Here, a part other than the voice section in FIGS. 13A and 13B is the ambient sound.

The ambient environment sound includes ambient environment noise other than voice, and voice not treated as a voice section because of low power or short duration.
Therefore, by examining the magnitude of the power of the ambient environment sound, the degree of quietness of the ambient environment such as buzz can be used as structural information. Since the information indicating the surrounding environment is different from the meaningful key data, it is defined here as environment information.

The ambient environment sound includes ambient environment noise other than voice, and voice that is not treated as a voice section because of low power or short duration.
Therefore, by examining the magnitude of the power of the ambient environment sound, the degree of quietness of the surrounding environment such as buzz can be used as structural information in the structural information.

By recognizing a characteristic sound that represents the atmosphere of the place, such as laughter and applause, as the ambient environment sound,
The atmosphere during use can be used as structural information for structural information. By combining the key data at the beginning and end of the voice section output by the voice section detector 81, the shorter the time from when one speaker finishes speaking until another speaker starts speaking. It is possible to judge that an active dialogue is taking place, and this can be used for structural information as environmental information. In FIG. 13 described above, the shorter the time from t2 when the speaker A finishes speaking to t3 when the speaker B begins speaking, the more active the conversation is.

Further, by creating a dictionary having standard patterns of reverberation characteristics, it is possible to estimate the place from the ambient sound. In particular, when the place where this device is used is limited to some extent, a dictionary can be created as a standard pattern of reverberation characteristics by collecting reverberation characteristics for each use location such as one's room, meeting room, corridor, and outdoors. . As shown in FIG. 14, the test sound generation unit 140 generates a system beep sound, which is output when the power is turned on, as a test signal and inputs it to the use location estimation unit 141 from a voice input device such as a microphone. The usage place estimation unit 141 collates with the usage place data stored in the reverberation characteristic dictionary 142.

The use place estimated in this way is used as environment information, and information indicating where the input is made is added and stored in the structure information storage unit 4 as structure information. In addition, the semantic content of the utterances exchanged in the dialogue is held as a dialogue history, and the current situation of the dialogue can be known from the semantic content of the newly input utterance and the dialogue history. It is also possible to use this as key data for structural information.

In this embodiment, the time-series data to be structurally analyzed is input via the time-series data input means 1. However, not only is this data analyzed online, but as shown in FIG. To (the same reference numerals as in FIG. 1),
The time-series data already stored in the time-series data storage unit 2 may be configured to be sent to the structure analysis unit 3, and the already-stored time-series data may be analyzed to generate new key data. Is.

By creating new key data,
The recognition process can be redone for the time series data that has already been analyzed. For example, set a word that is different from the word recognized at the time of data entry as a recognition vocabulary,
The structure can be analyzed again.

Further, in the case of not only word recognition but also meaning expression, when the word arrangement and meaning expression candidate range in the expected scene are exceeded, the user is required to use the word arrangement and meaning expression. Appropriate structural information can be generated by setting correctly and performing structural analysis again.

Next, some examples of key data obtained by using these key data will be shown. For simplicity, it is assumed that the voice data has already been separated for each speaker by, for example, preparing a microphone for each speaker.

It is assumed that the voice data for each speaker is analyzed, structure information is generated from the key data indicating the voice section, and stored in the structure information storage means 4. This key data is information indicating when a certain speaker uttered, and it is possible to know which speaker uttered when, how long, and how often. Also, how often the speakers changed, how many speakers uttered in a certain period of time, and the time difference between the end of one speaker and the start of another speaker. You can know how exciting conversations and discussions are based on the size of.

Similarly, by analyzing the voice data,
It is assumed that the structure information is generated from the key data indicating the recognition result of the word voice recognition and stored in the structure information storage means 4.
For example, when inputting voice data of meeting attendees, if the frequently occurring words for each topic in the meeting are used as the key data of the recognition target word, the topic and content of the discussion will be estimated from the word. I can know. Further, if a specific word spoken by a person is used as the key data of the recognition target word, it is possible to search for and extract a portion in which an important speech is made.

As described above, since the voice data already recorded in the time series data storage unit 2 can be analyzed, the recognition target word can be reset after the voice data is captured. It is also possible to search for important utterances that were not expected.

Further, it is assumed that the structure information is generated from the key data indicating the semantic expression of the utterance from the analysis of the voice data and stored in the structure information storage means 4. This makes it possible to know when and what topic was being discussed from the key data of the semantic expression about the approximate content. In addition, it is possible to know who spoke what and what was close to whom.
Also, if the transition of topics is illustrated, the flow of discussion can be arranged and illustrated / arranged.

When a plurality of time series data are stored,
If there is common structural information across these, it is possible to search by linking it. For example, if a plurality of voice data have a common speaker, the same speaker can be picked up.

When the voice data of a plurality of conferences are input, it is possible to retrieve the contents of a person who attended one conference from another conference from the structural information indicating the speakers. It is possible.

In addition, when there are words common to a plurality of voice data, it is necessary to search for a word that became a main topic in a certain meeting, how it was used in a previous meeting or not, by relying on structural information indicating the word. You can

Here, FIG. 16 shows an example in which who spoke what, what the transition of the topic, and the closeness of the contents of the speech are displayed on the screen. FIG. 16A shows the situation when a decision is taken in a conference held by five conference participants A, B, C, D, and E. Here, as a result of structural analysis of the voice data when the decision was taken, it represents the utterance meaning of who said what. A is for, B is for, C is for, D is for conditional, E is for It indicates that he has expressed his opinion.

FIG. 16B shows the progress of time in the conference and the closeness of the contents of the message. Here, there were plan 1 of D and plan 2 of B at the beginning, the contents of the speech were separated, and the conference was stalled without other speakers. However, the discussion was started after A issued the plan 3. It shows that D became more active and D became closer to A's opinion.

The structure information items are related to each other based on the structure information name or structure information ID and the time information, and new structure information is generated. The search, output, and user input of key data will be briefly described below.

As described above, the key data is used as the search key during the search. The user inputs the key data into the search command input means 5. Here, the word "yes" is input as key data. The key data input to the search command input means 5 is sent to the search means 6, and the search means 6 searches the structure information storage means 4 for all structural information having the key data of "yes". The search means 6 sends the searched structural information to the information output means 7, and the information output means 7 presents the searched structural information, and the user refers to this and displays the structural information.
Further, the key data of the structural information to be searched for is input using the search command input means 5.

The search command input means 5 further receives an input of a search key composed of key data, and sends the search command to the search means 6. For example, when playing back the time-series data when an affirmative statement is made from time t1 to t2, the start time "t
The user inputs key data “1”, end time “t2”, and utterance meaning “affirmation” into the search command input means 5.

The search means 6 searches the structure information storage means 4 for structure information that matches the key data entered by the user. Here, since the structural information “SR-5” shown in FIG. 12 is searched and the key data “Yes” has already been input, the structural information “SR-5” having the key data “Yes” and the structural information “SR-5” are input. The structural information "WD-7" having the key data "Yes" is retrieved from "."

When outputting only meaningful information, the retrieved structure information is sent to the information output means 7, and the information output means 7 outputs meaningful information which is a part of the sent structure information.

When outputting the voice data itself, the retrieval means 6 uses the ID of the original voice data in the already retrieved structure information "WD-7" to retrieve the voice data stored in the time series data storage means 2. Also access.

The information output means 7 is composed of a display device such as a speaker or a CRT, and reproduces the audio data which is the source of the retrieval by the retrieval means 6. Outputting the time-series data is equivalent to reproducing all or part of the input data, and outputting the structural information expresses the time-series data in a short time using a visual display, or hears the sound as a metaphor. Output. For example, in displaying the structural information obtained by word recognition, the recognized words are displayed using text or icons, and in displaying the structural information obtained by understanding the utterance, the meaning content by the text display of keywords is expressed. It is possible. In the display of the meeting record, icons are displayed and texts of the meaning contents are displayed according to the order of the seats of the speaker, the time of the talk, and the meaning contents of the talk. A record of what the other person said in response to the statement was displayed. In addition, FIG.
As shown in (b), the effective output can be performed by showing the information such as “the discussion is active” and “stagnation” in the visual expression such as the color tone and the shading.

The key data input means 8 is a mouse, a pen,
It is composed of a pointing device such as a touch panel or an input device such as a keyboard. The key data input means 8 accepts correction and addition of key data from the user by these input devices. When using these input devices, the key data and the structural information are displayed on the screen on the information output means 7, the key data to be corrected / added is shown at the cursor position, and the input from the user is the keyword input. If the means 8 exists, the corresponding structural information is changed.

In this way, by the user inputting the key data, not only the correction of the key data and the structure information automatically generated by the structure analysis means 3 but also the addition of the key data and the structure information by the user You can

The user inputs the key data while confirming the contents stored in the structure information storage means 4 and the time series data storage means 2 by the information output means 7,
It is possible to improve the structural information to meet the user's request,
The quality of structural information can be improved.

As one of the key data entered by the user, the user's evaluation of voice data and structural information can be included. For example, suppose that the evaluation of the utterance content of each speaker is divided into ranks such as ☆, ◯, Δ, and × and evaluated, and this is added to the structural information as key data. ☆ for utterances that seem to be extremely important, ○ for utterances that seem to be important, and △ for utterances that are not important but related to topic progression.
In particular, utterances that are not related to the progress of the topic are marked with x. If you want to find out the point that will be the point later, search for the ☆ part, if you want to summarize, search for the ○ part, and if you have time to follow the flow of the topic, search for the △ part. You can

Next, an example applied to a recognition / understanding evaluation system (hereinafter referred to as a recognition / understanding evaluation system) will be described. Generally, in order to improve the performance of the recognition / understanding system, it is necessary to collect a large amount of actual data actually used by the system, but it requires labor and labor to manage a large amount of collected data. As in this embodiment, all recognition / understanding results are obtained as key data, and if the link information with the time-series data from which they are obtained is held, each recognition / understanding result is obtained. Only time series data can be selectively output.

In this case, the input time series data is the input time series data to the recognition / understanding system and the recognition / understanding result data. When a plurality of processing steps are performed in the recognition / understanding system as in the above-described example, it is possible to handle them as time-series data including intermediate processing results in each step.

In the evaluation system, the evaluation scale for evaluating the recognition / understanding performance, that is, (A) desirable recognition /
The understanding result (interim result) or the correct recognition / understanding result (interim result) corresponds to the key data. Alternatively,
(B) The result of comparison between the recognition / understanding result of the recognition / understanding system (interim result) and the desired or correct recognition / understanding result (interim result) may be used as the key data. Alternatively, the key data may be (C) a label for time-series data that should be targeted in the future although it is not targeted for recognition / understanding by the current recognition / understanding system.

The link information may be (a) information indicating the correspondence with the processing result of the recognition / understanding system, or (b).
It may be information (for example, time, sample point, or data address) indicating the correspondence with the input time series data to the recognition / understanding system.

The above information can be input by the user. If the key data is used as the desired recognition result (A) and the link information is searched as (b), the actual data of the input data for a certain recognition target can be collected, and this can be used as learning data to create a more accurate recognition dictionary. Obtainable.

By summarizing the key data (B), the recognition / understanding performance can be obtained, and by tracing the link information, the time series data for a certain recognition target can be collected together with the correct / wrong judgment result.

By searching using the key data (C), it is possible to collect only actual data that cannot be handled by the current system. Less than,
An example in which the time-series data recording system of the present invention is applied to a voice dialog evaluation system for evaluating a dialog between a voice dialog system and its user will be described with reference to FIG.

As shown in FIG. 17, the voice dialogue evaluation system 170 is a system for recognizing and understanding the voice uttered by the user and returning a voice response or a screen display response to the user, and a voice dialogue system 171. Dialogue evaluation system 172 having functions such as accuracy of recognition result of system 171 and creation of dialogue voice database considering dialogue situation
Composed of. That is, here, the voice dialogue system 17
Reference numeral 1 is a structure analysis unit 3 which recognizes voice data and outputs structure information, and a dialogue evaluation system 172 is a time-series data recording / reproducing apparatus of the present invention, which is a voice dialogue system 171.
The speech recognition result of the speech dialogue system 171 is evaluated from the structural information input by the.

Hereinafter, the configuration of the dialogue evaluation system 172 is similar to that shown in FIG. Also, description will be given with the same reference numerals as those in FIG. The voice dialogue system 171 is an additional time series in which the time series data input means 1 of the dialogue evaluation system 172 (time series data recording / reproducing device) includes PCM data of dialogue voice and dialogue information such as recognition and understanding result and response content. Send data, key data, structural information.

FIG. 18 shows an example of a file output of dialogue information such as recognition and understanding results and response contents sent to the time series data input unit 1 of the dialogue evaluation system 172 by the voice dialogue system 171.
Shown in. The file output of this voice dialog system includes the utterance number indicating the number of utterances, the voice section of the voice dialog system and the user's utterance, the word detection result, the syntax,
It includes a plurality of semantic expression candidates obtained by the semantic analysis, a semantic expression (understanding result) selected from the plurality of candidates in consideration of the history of the dialogue, response contents, and the like.
The dialogue evaluation system 172 separates this input data into voice data and structural information.

In order to check the accuracy of the recognition / understanding result of the voice dialogue system 171, a human must give the correct key data of the word detection / recognition result and the understanding result of the meaning of the utterance. This is the dialogue evaluation system. The user additionally inputs using the key data input means 8 of 172.

FIG. 19 shows an example of a voice interactive system for accepting orders at a hamburger shop, and realizes a part of the key data input means 8 and the information output means 7 by using a window interface which is a window system library. This is an example, and the user of the dialog evaluation system can give the text of the correct word by clicking the icon of the recognized word displayed on the screen with the mouse or the pen. In this example, not only the recognized word, but 1
There is also a delimiter icon to indicate the end of the utterance. For example, if the key data input means 8 of the dialogue evaluation system 172 is configured by a window interface, the text of the correct recognition word can be easily input and modified. Here, when inputting the meaning of ordering one small size of orange juice, an icon 190 indicating “orange juice” and an icon 191 indicating “small” on the screen shown in FIG.
And an icon 192 indicating “one”, an icon 193 indicating “please” and an icon 194 indicating “utterance end”.
Touch to enter.

By the way, the text of the correct recognition word input through the window interface of FIG. 19 does not include time information such as the start and end of the word. In order to obtain the time information, the user of the dialogue evaluation system 172 decides the dialogue voice data from the information output means 7 by partially reproducing and outputting the section in which the recognized word exists and listening and checking. The section in which the recognized word exists is the dialogue evaluation system 17
2 users input and decide, but voice dialogue system 1
If the start and end of the recognized word obtained as a result of recognition by 71 are set as initial values, the burden of input on the user of the dialog evaluation system 172 is significantly reduced.

Even when the correct answer of the semantic expression of the utterance is input,
This can be achieved by preparing a window interface. Further, if the correct word sequence has already been obtained, the semantic expression of the utterance content can be obtained by performing the semantic analysis. By modifying this semantic expression, the data that the user of the evaluation system needs to directly input is reduced.

The structural information obtained by inputting the correct key data by the user in this way is the voice interaction system 17
1 is output, that is, the key data and structure information input to the dialog evaluation system 172 are collated, and the structure information indicating the word detection performance of the voice dialog system 171 is constructed.

For example, the user inputs the key data “Yes” of the correct word with the key data input means 8,
Structural information indicating the correct word shown in (a) is created.
Based on the time information of the structure information “WC-5” indicating the correct word shown in FIG. 20A, that is, the start time “t1” and the end time “t2”, the structure analysis unit 3 performs the structure shown in FIG. By comparing with the structural information showing the word detection result shown in, it is checked whether a correct word is detected, whether there is a word that is not detected (dropped), or whether a word that is not in the correct answer is detected (insert),
Structural information indicating the word detection performance is generated from each result.

Here, the structure information “WD-7” in FIG.
20B, the pointer or structure information ID of the structure information “WD-7” indicating the word detection result is included and combined with the information of the structure information “WC-5” indicating the correct word, and is shown in FIG. Such structure information that indicates the word detection performance of the voice interaction system 171 can be created.

Further, the structure information indicating the word detection performance is collated with the structure information indicating the word recognition result to generate the structure information indicating the word recognition performance. Here, as described above, since the structure information indicating the word recognition result has the information linked to the structure information indicating the word detection result, it can be determined whether the word is the correct answer or not. Therefore, it is possible to determine whether or not each of the word strings included in the word recognition result is correct. In this way, the voice dialogue system 17
The word recognition performance of 1 can be evaluated.

The structural information indicating the semantic expression of the correct utterance is collated with the semantic expression candidate and the structural information indicating the selected semantic expression to determine whether the semantic expression candidate to be selected is correctly selected or the semantic expression candidate. The structural information indicating the comprehension performance including information about whether or not there is something to be selected is generated.

The above-mentioned structural information can be used for improving the performance of the voice dialogue system 171 as described below. For example, the improvement of word detection performance will be described. From the structural information indicating the correct word input by the user of the dialogue evaluation system 172, the word voice data in the actual dialogue can be listed. This makes it possible to obtain a voice database that collects voice data of only certain words. If the dictionary for word detection is relearned using this voice data, the word recognition performance can be improved. Further, if the structural information indicating the word detection performance is used, the detection accuracy for each word can be obtained, and if the learning is performed again from a word having a low detection performance, the performance can be rapidly improved.

Information output means 7 of the dialogue evaluation system 172
In FIG. 21, an output example of displaying the structural information in association with the change of the topic and the temporal passage of the dialogue will be described with reference to FIG. Here, the file output example of FIG. 18 is displayed.

The conversation voice waveform display section 210 is a voice waveform of conversation voice. This is a reproduction of time series data of voice in a waveform. The text display unit 211 outputs time-series data with time information expressed in text.

The word voice waveform display section 212 outputs the voice waveform of the blinking portion of the conversation content display section 211. The word detection result display unit 213 displays words detected from the voice waveform displayed by the word voice waveform display unit 212. Here, words having similar waveforms, words having similar voice intervals, etc. are detected.

The word recognition result display unit 214 displays the word recognized as correct from the detected words. Here, the word uttered from 4.74 seconds to 5.12 seconds is "potato".
Has been recognized.

The syntax-meaning-candidate display unit 215 displays candidates for the meaning of a sentence from the detected words and the recognized words. Here, five candidates that mean an order are output from the detected words, but the first candidate is selected from the recognized words.

The syntax-meaning-candidate information display unit 216 displays the meaning of ordering “two hamburgers”, “one large potato”, and “three colas” from the candidates selected in the syntax-meaning candidate display unit 215.

The syntactic and meaning correction information display section 217 displays the meaning and candidate information corrected when the meaning and candidate information displayed on the syntactic and meaning candidate information displaying section 216 contains an error. As described above, by using the structural information related to the dialogue history and displaying it in association with the transition of the topic and the passage of time of the dialogue, it is possible to visually display whether the dialogue is stagnant or smoothly progressing. .

The handling of the evaluation dialogue data (time series data) in the above "speech dialogue evaluation system" is summarized below. Here, the time-series data are: -Voice data at the time of dialogue (at least 2ch of system response and user utterance).

Then, as what can be treated as both time-series data and structure information, for example, as the recognition result of the system shown in FIG. 18, system response, user's utterance start / end time (generally plural・ The (word) voice recognition result extracted from the user's utterance (including the start time and end time of the word) ・ User's utterance meaning understanding result (for one utterance) ・ System internal state ・ System response contents (these are These are almost equivalent to the processing results of the voice section detecting section, the voice recognizing section (word detecting section), the voice understanding section, the dialogue managing section, and the response generating section of the "speech dialogue system", and "the desirable processing". There is a "correct answer" entered by the system developer (evaluator).

Then, for evaluation, "spoken dialogue system"
And a user dialogue using voice (using screen display). When the user's utterance is input to the voice dialogue system, the voice data, the start / end time of the utterance, the voice recognition result, the utterance meaning understanding result, are analyzed and recognized by the voice data, the dialogue process, and the response generation. The internal state and response contents are determined and generated respectively.

The "speech dialogue evaluation system" records each of the above-mentioned processing data and processing results in order to improve the system performance and the user interface, and each "input by the evaluator (user of the speech dialogue evaluation system)" is recorded. Information useful for system improvement such as performance evaluation information of the current "speech dialogue system" or link information with the input data (time series data) that is the source of each processing compared with "desired processing result" I will provide a.

Here, as an example of evaluating the performance based on the obtained information, the voice recognition performance is the voice of the “correct answer” input by the system developer as “desired processing” and the processing result of the system. Evaluated by comparing the recognition results ・ The speech understanding performance is evaluated by comparing the "correct answer" entered by the system developer as "desired processing" and the utterance meaning understanding result with the system processing result, etc. And "desirable treatment" are compared.

As an example of using the obtained information as useful information for improvement, in order to improve the voice recognition performance, the learning data of the recognition dictionary in which the correct dialogue result label is attached to the actual dialogue voice data is used. Can be output. You can also list new vocabulary to be added.・ In order to improve the voice comprehension performance, you can list unexpected utterances and add grammar to improve comprehension performance.・ It is possible to list the detection error of the voice section in order to improve the voice section detection performance.

As described above, by accumulating the structural information which is the recognition result of each recognizing means and improving the performance of each recognizing means, it is possible to improve the performance of the entire voice dialogue evaluation system.

Although the example of the audio data has been described as the time series data, an example of handling the image data as the time series data will be briefly described below. Image data is input from the time-series data input means 1 including a camera or the like.
The time series data input means 1 adds identification data to the image data and sends it to the time series data storage means 2 and the structure analysis means 3. The time-series data storage means 2 composed of a device for recording image data such as a VTR device records the image data to which the sent identification data is added.

The structure analysis means 3 for recognizing an image is
Detect key data and generate structural information. With the current image recognition apparatus, it is possible to recognize the "human being has moved from time ta to time tb". Therefore, it is possible to detect the key data of “human” and “movement”.
Further, structural information such as “ta to tb as time information, human as recognition result, ID of image data” or “ta to tb as time information, move as recognition result, I of image data I
Structural information such as "D" can be created.

When searching the image data, key data such as “human” and “movement” are searched for by the search command input unit 5.
By inputting in
The structural information having the key data "move" is searched, and the image data linked with this structural information is searched.
The information output means 7 reproduces the searched image data.

Further, the key data input means 8 is used to
By specifically changing the key data to a person's name, more accurate key data and structure information can be generated.

In the case of image data with audio data such as a movie or a video, key data for audio is detected and structural information is generated, and key data for image is detected and structural information is generated and utilized. By doing so, a high quality database can be realized.

[0130]

As described above, according to the present invention, in recording / reproducing time-series data with multimedia data as an input, the time-series data is recognized, key data is searched, and generated from the key data. The structural information can be recorded together with the time series data, and the user can search based on the key data to reproduce only the time series data or information meaningful to the user. Also, by revising and adding key data, structural information is regenerated,
We can provide a high-quality database for users that is closer to their needs. Further, the recognition / understanding can be evaluated based on the structure information of the time series data, and the system performance can be improved.

[Brief description of drawings]

FIG. 1 is a diagram showing an overall configuration in an embodiment of the present invention.

FIG. 2 is a diagram showing a procedure from input of time series data to storage in one embodiment of the present invention.

FIG. 3 is a diagram showing a procedure of searching for stored data and outputting the data according to an embodiment of the present invention.

FIG. 4 is a diagram showing additional time series data to which a data type and time information are added according to an embodiment of the present invention.

FIG. 5 is a diagram showing additional time-series data corresponding to address information indicating a storage position of time-series data in an embodiment of the present invention.

FIG. 6 is a diagram showing an example of structural information according to an embodiment of the present invention.

FIG. 7 is a diagram showing structure information corresponding to address information indicating a storage position of structure information according to an embodiment of the present invention.

FIG. 8 is a diagram showing an internal configuration of a structural analysis means 3 in one embodiment of the present invention.

FIG. 9 is a diagram showing a graph of a relationship between time change and voice power in one embodiment of the present invention.

FIG. 10 is a diagram showing structural information about detection of a voice section in the embodiment of the present invention.

FIG. 11 is a diagram showing structural information about word detection according to an embodiment of the present invention.

FIG. 12 is a diagram showing structure information related to candidates of semantic content of utterances in an example of the present invention.

FIG. 13 is a diagram showing the relationship between the time change and the voice power of two speakers according to an embodiment of the present invention.

FIG. 14 is a diagram showing a configuration of a system for estimating a place of use in an embodiment of the present invention.

FIG. 15 is a block diagram configured so that already stored time series data in one embodiment of the present invention can be sent to the structure analysis means 3.

FIG. 16 is a diagram in which transitions of utterance contents and topics and closeness of utterance contents are displayed on a screen according to an embodiment of the present invention.

FIG. 17 is a configuration diagram when applied to a voice dialogue evaluation system in one embodiment of the present invention.

FIG. 18 is a diagram showing an example of a file output of dialogue structural information such as a result of recognition and understanding of time-series data and response contents in an embodiment of the present invention.

FIG. 19 is a diagram showing an example in which the key data input means in one embodiment of the present invention is realized by using a window interface.

FIG. 20 is a diagram showing structural information about correct words in an example of the present invention.

FIG. 21 is a diagram showing an example in which structural information in one embodiment of the present invention is displayed in association with topic changes and the temporal passage of dialogue.

[Explanation of symbols]

 1 time-series data input means 2 time-series data storage means 3 structure analysis means 4 structure information storage means 5 search command input means 6 search means 7 information output means 8 key data input means 81 voice section detection section 82 acoustic analysis section 83 word detection Part 84 Syntax Semantic Analysis Part 85 Environment Information Extraction Part 140 Test Sound Generation Part 141 Use Location Estimation Part 142 Reverberation Characteristic Dictionary 170 Spoken Dialogue Evaluation System 171 Spoken Dialogue System 172 Dialogue Evaluation System 190, 191, 192, 193, 194 Icon 210 Conversation Speech waveform display unit 211 Text display unit 212 Word speech waveform display unit 213 Word detection result display unit 214 Word recognition result display unit 215 Syntax meaning candidate display unit 216 Syntax meaning candidate information display unit 217 Syntax meaning correction information display unit

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Yoichi Takebayashi, 1 Komukai Toshiba-cho, Sachi-ku, Kawasaki City, Kanagawa Prefecture, Research & Development Center, Toshiba Corp. Town No. 1 In stock company Toshiba Research and Development Center (72) Inventor Hiroshi Mizoguchi No. 1 Komukai Toshiba Town, Kouki-ku, Kawasaki City, Kanagawa Prefecture Inside company Toshiba Research and Development Center (72) Inventor Hisako Tanaka Yuko Kawasaki, Kanagawa Prefecture Komukai-shi, Toshiba-cho 1 Incorporated company Toshiba Research and Development Center (72) Inventor Hideaki Shinchi 1385 Shinmachi, Ome-shi, Tokyo Inside Toshiba Software Engineering Co., Ltd.

Claims (6)

[Claims] 1. A time-series data recording / reproducing apparatus for storing time-series data and reproducing the stored time-series data, wherein key data is detected from input time-series data, and at least the detected key data and Structural information analysis means for generating structural information composed of key data and link information for associating the time series data, structural information storage means for storing the structural information generated by the structural information analysis means, and the key data Using the search key as a search key, the structure information stored in the structure information storage means is searched, and the time series data is searched based on the link information of the searched structure information. A time-series data recording / reproducing apparatus, which reproduces the time-series data retrieved by. 2. The time-series data recording / reproducing according to claim 1, wherein when reproducing the time-series data retrieved by the retrieval means, only a predetermined portion of the time-series data including the key data is reproduced. apparatus. 3. A time-series data recording / reproducing apparatus for storing a plurality of time-series data and reproducing the stored time-series data, wherein key data is detected from the input time-series data, and at least the detected key data and A structure information analysis unit for generating structure information composed of the key data and link information for associating the time-series data; a structure information storage unit for storing the structure information generated by the structure information analysis unit; A search for searching the structure information stored in the structure information storage means using key data as a search key, and searching for predetermined time series data from a plurality of time series data based on link information of the searched structure information. Means for reproducing the predetermined time series data retrieved by the retrieval means. . 4. Key data is detected from input time-series data, and at least the detected key data, link information for associating the key data with the time-series data, and environment information representing the environment of the time-series data are configured. Structure information analyzing means for generating structure information, structure information storing means for storing structure information generated by the structure information analyzing means, and the structure information storing means for storing the key data as a search key. A search unit that searches the structural information and searches for environmental information of a portion corresponding to the key data of the searched structural information, and the environmental information searched by the searching unit is visible to the user. A time-series data recording / reproducing device characterized by reproducing in a form. 5. The time-series data recording / reproducing apparatus according to claim 1, further comprising key data input means for a user to directly input key data in order to create new structural information. 6. A time-series data recording / reproducing apparatus according to claim 1, further comprising key data input means for allowing a user to directly input structural information.