JP4600828B2 - Document association apparatus and document association method - Google Patents

Description

  The present invention relates to a document association apparatus and a document association method, and in particular, a document association apparatus and a document association method for deriving a correspondence relationship between content such as video or audio and document information related to the content. About.

  Methods are known for automatically mapping document data to corresponding parts of an audio recording or an accompanying video recording. For example, in Japanese Patent Application Laid-Open No. 7-199379, the voice in the voice recording or the video recording accompanying the voice is converted into text by voice recognition processing, and the text is compared with the document information stored in order in the document storage device. A method has been proposed in which both are considered identical if they contain the same series of characters. At this time, the automatic speech recognition device decodes the speech, and the decoded text is collated with the document information through identification of similar words or clusters of words.

  In Japanese Laid-Open Patent Publication No. 2000-270263, when an announcement manuscript and subtitle content are very similar in a broadcast program, the announcement manuscript is subjected to voice recognition processing and arranged in order of voice recognition result and presentation time. There has been proposed a system that detects and records the timing information of the start point and the end point as a synchronization point by deriving the correspondence with the closed caption text.

  Furthermore, in Japanese Patent Application Laid-Open No. 8-212190, when a scenario text is associated with a voice-accompanying moving image, a silent section when the scenario text is voiced is predicted, and the prediction result and the voice signal in the voice-accompanying moving image A system for associating speech and text by comparing with silent sections has been proposed.

  The first problem with the conventional document association method for associating content such as video or audio with a document is that the accuracy of correspondence between audio information and document data depends greatly on the accuracy of speech recognition. This means that when the accuracy of recognition is not sufficiently obtained, the correspondence relationship between the voice information and the document data cannot be accurately derived.

  In the conventional methods described in JP-A-7-199379 and JP-A-2000-270263, after converting speech into text by speech recognition processing, synchronization between the converted text and document data is performed. I'm taking it. As a result, if the text output by speech recognition contains many errors, there are many errors in the correspondence, such as failure to correspond to document data or correspondence to completely different document parts. It will be included. In general, in speech recognition, recognition accuracy may be significantly reduced when BGM is superimposed on speech, or when background sounds other than spoken speech are loud, such as conversations recorded under high noise such as outdoors. Are known. Even in normal conversation, there are many cases where high recognition accuracy cannot be expected due to the positional relationship between the microphone and the speaker, the way the speaker speaks, the conversation style and characteristics, and the like. When the conversation content is limited to a specific topic, it is possible to improve the accuracy of speech recognition by taking measures such as optimally selecting a dictionary for recognition based on the estimated topic. However, in general, topics and the like cannot often be estimated in advance, and in that case, there is a problem that the accuracy of speech recognition is further lowered if an incorrect dictionary is used. Based on these speech recognition results containing many errors, when correspondence between voice recording or video recording accompanied by voice and document information is increased, the number of correspondence errors increases, and text search or keyword search is performed. It becomes difficult to use it.

  The second problem with the conventional method is that if the document information is not a faithful reproduction of the sound but is a document that simply summarizes the contents, the document information and the sound information cannot be correctly aligned. There is. For example, when associating speech information in a lecture with explanatory materials or summary documents created by a speaker, there is no portion in the document that directly corresponds to the text created from the speech information. Cannot be properly aligned.

  As a third problem of the conventional method, in the matching based on speech recognition, the unit of matching is a word unit. Therefore, when the document content and the speech information do not completely match, This means that the correspondence will be greatly shifted by the appearance.

  As a related technique, Japanese Patent Application Laid-Open No. 2000-348644 (priority claim number: 09/288724, priority claim country: US) discloses a method for searching speech information using content information and speaker information, and An apparatus is disclosed. The method for retrieving speech information using the content information and the speaker information is a method for retrieving speech information from one or more speech sources. Receiving a user query specifying at least one content and one speaker constraint; and comparing the user query with a content index and a speaker index of the speech source to obtain speech information that matches the user query. Identifying.

  As a related technique, Japanese Unexamined Patent Application Publication No. 2002-189728 discloses a multimedia information editing apparatus, a method thereof, a recording medium, and a multimedia information distribution system. This multimedia information editing apparatus edits multimedia information. The multimedia information editing apparatus includes a storage unit, a voice discrimination unit, a document conversion unit, and a multimedia structuring unit. The storage means stores multimedia information such as voice and moving images. The voice discrimination means determines whether voice is added to the multimedia information stored in the storage means. The document conversion means converts the voice information into document information when the voice is added by the voice discrimination means. The multimedia structuring means linguistically analyzes the document converted by the document conversion means to structure and associate the document with the multimedia information.

As a related technique, Japanese Patent Application Laid-Open No. 2002-236494 discloses a technique of a speech segment determination device, a speech recognition device, a program, and a recording medium. This speech segment determination device is characterized by comprising acoustic analysis means, standard pattern storage means, matching means, determination means, and speech segment determination means. The acoustic analysis means acoustically analyzes a voice input from the outside with a predetermined period, and obtains an acoustic feature amount based on the analysis result. The standard pattern storage means stores a standard pattern corresponding to the voice of a single speaker and the mixed voice of a plurality of speakers on the premise that voices of a plurality of speakers can be mixed in the input voice. The matching unit performs matching between the standard pattern stored in the standard pattern storage unit and the acoustic feature amount obtained by the acoustic analysis unit.
The determination unit determines, based on the processing result by the matching unit, which standard pattern the input speech is similar to every predetermined period. The voice section determining means comprises determining the voice section of each speaker based on the determination result by the determining means.

  As a related technique, Japanese Patent Application Laid-Open No. 2002-366552 (priority claim number: 09/966659, priority claim country: US) discloses a method and system for searching recorded speech and searching for related segments. . This is a method for searching recorded voices in a database. a) converting the recorded speech into text using a speech recognition system; and b) creating a full text index of the recorded speech using an information extender, wherein the full text index is Including a plurality of time stamps indicating the occurrence of a word in the recorded speech, c) searching the text using the full text index by a full text server; and d) the search text; Storing the full text index and the recorded audio in the database. The specific content of the recorded audio is reproduced using the full text index without listening to the entire recording.

  As a related technique, Japanese Patent Application Laid-Open No. 11-242669 discloses a technique of a document processing apparatus. The document processing apparatus includes a voice input unit, an extraction unit, an attribute generation unit, a document storage unit, an instruction unit, an output unit, and an attachment unit. The voice input means inputs voice. The extraction means extracts information for specifying a speaker from the voice input by the voice input means. The attribute generation means generates speaker attribute information by comparing the extracted information with predetermined reference information. The document storage means stores the document. The instruction means indicates a position in the document to which the input voice is to be attached. The output means outputs a document. The attachment means stores, in the document storage means, set information including position information in the document instructed by the instruction means, the input voice, and speaker attribute information generated by the attribute generation means. To do.

  An object of the present invention is to provide a document associating apparatus and a document associating method for accurately associating a significant section defined in content such as audio and video with a section in a document.

  Another object of the present invention is to provide a document associating apparatus and a document associating method that accurately associates a significant section in a content with a section in a document without being affected by the state of the content.

  Another object of the present invention is to provide a document associating apparatus and a document associating method that accurately associates a significant section in content and a section in a document without being affected by the type of document.

  These objects and other objects and advantages of the present invention can be easily confirmed by the following description and attached drawings.

  In order to solve the above-described problem, the document association method of the present invention includes (a) content including at least one of audio information and video information in which a plurality of speakers appear as speakers, and the content of the content. Preparing a described document; and (b) deriving a correspondence between the content and the document for each speaker.

  In the document association method, the step (b) includes: (b1) dividing the content into units of speakers to form a plurality of content sections; and (b2) dividing the document into units of speakers. A plurality of document sections; and (b3) a step of associating the plurality of content sections with the plurality of document sections.

  In the document association method, the step (b2) includes (b21) extracting from the content a point in time when the speaker changes from one of the plurality of speakers to another one of the plurality of speakers; (B22) dividing the content into units of speakers based on the time when the speaker changes.

  In the document association method, the step (b21) includes a step (b211) in which the content is the voice information and the change point of the voice of the speaker is extracted from the voice information.

  In the document association method, the step (b21) includes a step (b212) in which the content is the video information and the change point of the video of the speaker is extracted from the video information.

  In the document association method, the content is audio / video information in which the audio information and the video information are synchronized.

  In the document association method, the step (b21) includes the step (b213) of analyzing the change point of the sound feature of the voice information and deriving the time point when the speaker changes.

  In the document association method, the step (b21) includes a step (b214) of performing a change point analysis of the visual feature of the video information and deriving a time point when the speaker changes.

  In the document association method, the step (b21) includes (b215) performing a change point analysis of the visual feature of the video information and a change point analysis of the sound feature of the audio information, and integrating both results. Deriving a point in time when the speaker changes.

  In the document association method, the step (b) includes the step (b4) of analyzing the structure of the document and dividing the document into units of speakers.

  In order to solve the above problems, the computer program product of the present invention comprises program code means for executing all the steps described in any one of the above items when used on a computer.

  The computer program product having the above program code means is stored in a storage means readable by a computer.

  In order to solve the above problems, a document association apparatus of the present invention includes a content section extraction unit, a document section extraction unit, and a section correspondence relationship deriving unit. The content section extraction unit divides the content for each speaker and extracts a plurality of content sections for content including at least one of audio information and video information in which a plurality of speakers appear as speakers. The document section extraction unit extracts a plurality of document sections by dividing a document describing the contents of the content into units of speakers. The section correspondence relationship deriving unit derives a correspondence relationship between the plurality of content sections and the plurality of document sections.

  In the document association apparatus, the content is the audio information. The content section extraction unit analyzes the sound feature of the audio information and extracts the plurality of content sections.

  In the document association apparatus, the content is the video information. The content section extraction unit analyzes the visual feature of the video information and extracts the plurality of content sections.

  In the document association apparatus, the content is audio / video information in which the audio information and the video information are synchronized. The content section extraction unit extracts the plurality of content sections by integrating the result of the sound feature analysis of the audio information and the result of the visual feature analysis of the video information.

  In the document association apparatus, the content extraction unit includes an audio segment extraction unit, a video segment extraction unit, and an audio / video segment integration unit. The speech segment extraction unit analyzes the sound feature of the speech information, divides the speech information into speaker units, and extracts a plurality of speech segments. The video section extraction unit analyzes the visual characteristics of the video information, divides the video information into units of speakers, and extracts a plurality of video sections. The audio / video segment integration unit extracts the plurality of content segments based on the plurality of audio segment information regarding the plurality of audio segments and the plurality of video segment information regarding the plurality of video segments.

  In the document association apparatus, the content section extraction unit extracts a speaker change point as a point when the speaker changes from one of the plurality of speakers to another one of the plurality of speakers in the content. Then, the plurality of content sections are extracted.

  In the document association apparatus, the content includes the audio information. The content section extraction unit extracts the speaker change point based on a change in the feature of at least one prosodic information among the speech height, speech speed, and speech size in the speech information.

  In the document association apparatus, the content includes the audio information. The content section extraction unit extracts the speaker change point based on the change of the conversation form in the voice information.

  In the document association apparatus, the content includes the video information. The content section extraction unit extracts the speaker change point based on the change in the visual feature of the person in the video information.

  In the document association apparatus, the content includes the video information. The content section extraction unit extracts the speaker change point based on the change in the facial feature of the person in the video information.

  In the document association apparatus, the content includes the video information. The content section extraction unit extracts the speaker change point based on the change in the visual feature of the clothes of the person in the video information.

  In the document association apparatus, the document section extraction unit extracts the plurality of document sections based on the format information of the document.

  In the document association apparatus, the document section extraction unit extracts the plurality of document sections based on the description about the speaker entered in the document.

  In the document association apparatus, the document section extraction unit extracts the plurality of document sections based on the tag information of the structured document in the document.

  In the document association apparatus, the document section extraction unit extracts the plurality of document sections based on a change in conversation characteristics in the document.

  In the document association apparatus, the section correspondence relationship deriving unit is configured to compare the plurality of content sections and the plurality of documents based on the comparison between the section lengths of the plurality of content sections and the document amounts of the plurality of document sections. Correlate with the section.

  In the document association apparatus, the section correspondence relationship deriving unit performs the association based on the execution result of the dynamic programming matching for the plurality of content sections and the plurality of document sections.

  In the document association apparatus, the section correspondence relationship deriving unit includes a content speaker identifying unit, a document speaker information extracting unit, and a section matching unit. The content speaker identification unit identifies a speaker in at least one of the plurality of content sections. The document speaker information extraction unit specifies a speaker in at least one of the plurality of document sections, and obtains speaker information as information of the speaker. The section matching unit performs matching between the plurality of content sections and the plurality of document sections based on the speaker information.

  In the document association apparatus, the content speaker identification unit includes a content feature amount extraction unit, a speaker information storage unit, and a feature amount matching identification unit. The content feature amount extraction unit extracts a feature amount in at least one of the plurality of content sections. The speaker information storage unit stores the feature quantity and the speaker in association with each other. The feature quantity matching identification unit identifies the speaker based on the comparison between the stored feature quantity and the extracted feature quantity.

In the above document matching apparatus,
The content speaker identification unit identifies the speaker based on the characteristics of at least one of the prosody information of the voice pitch, voice length, and voice strength in the voice information. .

  In the document association apparatus, the content speaker identification unit identifies the speaker based on the feature amount represented by the conversation form in the audio information.

  In the document association apparatus, the content speaker identification unit identifies the speaker based on the visual feature amount of the person in the video information.

  In the document association apparatus, the content speaker identification unit uses a facial feature of a person as a visual feature of the person.

  In the document association apparatus, the document speaker information extraction unit specifies the speaker based on the description about the speaker entered in the document.

  In the document association apparatus, the document speaker information extraction unit specifies a speaker based on the metadata of the structured document in the document.

  In the document association apparatus, the section matching unit includes the plurality of content sections and the plurality of content sections so that a speaker in each of the plurality of content sections matches a speaker in each of the plurality of document sections. Is associated with the document section.

  In the document association apparatus, the section matching unit determines the plurality of content sections and the plurality of document sections based on the execution result of the dynamic programming matching for the plurality of content sections and the plurality of document sections. Associate.

  In the document association apparatus, the content includes audio information. The document association apparatus further includes a voice recognition unit that extracts utterance contents in the plurality of content sections and outputs utterance text information. The section correspondence relation deriving unit associates the plurality of content sections with the plurality of document sections based on the similarity between the utterance text information and the document information of the document.

  In the document association apparatus, the section correspondence relationship deriving unit is configured to determine the utterance text information based on the execution result of the dynamic program matching between the words that appear in the utterance text information and the words that appear in the document information. And its document information.

  In the document association apparatus, the section correspondence relationship deriving unit includes a basic word extracting unit and a basic word group similarity deriving unit. The basic word extraction unit includes one or more first basic words used in each of the plurality of content sections in the utterance text information, and one or more used in each of the plurality of document sections. Are extracted. The basic word group similarity deriving unit measures the similarity between the plurality of first basic words and the plurality of second basic words. The section correspondence relationship deriving unit derives the correspondence relationship based on the similarity.

  In the document association apparatus, the section correspondence relationship deriving unit derives the correspondence relationship by associating the similarity by dynamic programming matching.

  In the document association apparatus, a content input unit for inputting the content, a content storage unit for storing the content, a document input unit for inputting the document information, a document storage unit for storing the document, And an output unit that outputs information on the correspondence relationship.

  According to the present invention, even when the accuracy of speech recognition cannot be sufficiently obtained due to the influence of BGM, the influence of noise, the utterance style of the speaker, the sound collection environment, etc. It is possible to associate sections in the document. This is because the content such as voice or video and the document section are matched based on the unit of speaker (the part where the speaker has changed), which is easier than voice recognition. Recognizing the point where the speaker has changed is more robust against noise and sound collection because it only needs to recognize the difference than recognizing what the speaker is speaking. In addition, visual information can also be used because the correspondence is focused on the speakers rather than the contents of the voice, and if the speaker change points are extracted based on the visual information, the sound collection state Correspondence that does not depend on can be performed. Further, according to the present invention, it is possible to perform the association even when the document to be associated does not faithfully represent the conversation in the audio or video. The reason for this is that since matching at the word level is not achieved, it is possible to realize association in a relatively long section for each speaker or topic, and it is not necessary to associate the contents of individual conversations in detail.

FIG. 1 is a diagram showing a configuration of an embodiment of a document association apparatus according to the present invention. FIG. 2 is a block diagram showing an example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 3 is a flowchart showing an example of the operation of the content section extraction means 5 in the embodiment of the document association method of the present invention. FIG. 4 is a block diagram showing another example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 5 is a flowchart showing another example of the operation of the content section extraction means 5 in the embodiment of the document association method of the present invention. FIG. 6 is a block diagram showing still another example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 7 is a flowchart showing still another example of the operation of the content section extracting means 5 in the embodiment of the document association apparatus of the present invention. FIG. 8 is a block diagram showing another example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 9 is a flowchart showing another example of the operation of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 10 is a flowchart showing an example of the operation of the document section extraction means 6 in the embodiment of the document association apparatus of the present invention. 11A to 11D are diagrams illustrating an example of a method of using document format information in the embodiment of the document association method of the present invention. FIG. 11B is a diagram showing an example of a method of using document format information in the embodiment of the document association method of the present invention. FIG. 11C is a diagram showing an example of a method of using document format information in the embodiment of the document association method of the present invention. FIG. 11D is a diagram showing an example of a method of using document format information in the embodiment of the document association method of the present invention. FIG. 12A is a diagram showing another example of the method using the document format information in the embodiment of the document association method of the present invention. FIG. 12B is a diagram showing another example of the method using the document format information in the embodiment of the document association method of the present invention. FIG. 12C is a diagram showing another example of the method using the document format information in the embodiment of the document association method of the present invention. FIG. 13 is a diagram showing still another example of a method of using document format information in the embodiment of the document association method of the present invention. FIG. 14 is a block diagram showing an example of the configuration of the section correspondence relationship deriving means 7 in the embodiment of the document association apparatus of the present invention. FIG. 15 is a flowchart showing an example of the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention. FIG. 16A is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 16B is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 17 is a diagram for explaining normalization in the correspondence derivation method. FIG. 18A is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 18B is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 19 is a block diagram showing another example of the configuration of the section correspondence relationship deriving means 7 in the embodiment of the document correspondence apparatus of the present invention. FIG. 20 is a flowchart showing another example of the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention. FIG. 21 is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 22 is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 23 is a block diagram showing another example of the configuration of the section correspondence derivation means 7 in the embodiment of the document association apparatus of the present invention. FIG. 24 is a block diagram illustrating an example of the configuration of the candidate text document corresponding unit 62. FIG. 25 is a flowchart showing another example of the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention. FIG. 26 is a diagram showing the correspondence between content information and document information in the correspondence derivation method. FIG. 27 is a diagram illustrating the correspondence between content information and document information in the correspondence derivation method.

  Embodiments of a document association apparatus and a document association method of the present invention will be described below in detail with reference to the accompanying drawings.

The configuration of the embodiment of the document association apparatus of the present invention will be described.
FIG. 1 is a diagram showing a configuration of an embodiment of a document association apparatus according to the present invention. The document association apparatus 10 includes a content input unit (content input unit) 1, a document input unit (document input unit) 2, a content storage unit (content storage unit) 3, and a document storage unit (document storage unit) 4. , Content section extraction means (content section extraction section) 5, document section extraction means (document section extraction section) 6, section correspondence relation derivation means (section correspondence relation derivation section) 7, output means (output section) 8, It comprises. The content input unit 1 inputs content including information (data) such as audio and video. The document input unit 2 inputs a document related to the content. The content storage unit 3 stores the content obtained from the content input unit 1. The document storage unit 4 stores the document obtained from the document input unit 2. The content section extraction means 5 extracts a single speaker section from the content. The document segment extraction means 6 extracts a single speaker segment from the document. The section correspondence relationship deriving unit 7 derives a correspondence relationship between the content section extracted by the content section extracting unit 5 and the document section extracted by the document section extracting unit 6. The output unit 8 outputs the correspondence relationship derived by the section correspondence relationship deriving unit 7.

  The content input means 1 is for inputting target content. The content input unit 1 is, for example, a video camera or a microphone. Here, the content is exemplified by video information, audio information, or video information accompanied by audio information. The content input unit 1 may be a video player or a recording / reproducing device that reads and outputs video information or audio information recorded on a recording medium such as a video tape.

  The document input means 2 is for inputting a document related to the content. The document input unit 2 is a text input device such as a keyboard, a pen input device, or a scanner. The document input unit 2 may be an input device that reads document data created using document creation software.

  The content storage unit 3 is, for example, an internal storage device or an external storage device that records content from the content input unit 1. The storage medium used in the content storage unit 3 is exemplified by a RAM, a CD-ROM, a DVD, a flash memory, and a hard disk.

  The document storage unit 4 is an internal storage device or an external storage device that records a document from the document input unit 2. The recording medium used in the document storage unit 4 is exemplified by a RAM, a CD-ROM, a DVD, a flash memory, and a hard disk.

  The content section extraction means 5 divides the content (information) stored in the content storage means 3 into sections for each speaker, and extracts a content section by a single speaker. A content section by a single speaker (hereinafter, also referred to as “single speaker section”) is a section from when the speaker changes until the next speaker changes. The single speaker section is extracted so that there is a single speaker in the section and the speakers in the adjacent sections are different. The single speaker section extracted by the content section extraction unit 5 preferably does not include an error, but may include an error because the content section extraction is automated.

  The document section extraction unit 6 extracts a section (document section) corresponding to each speaker from the document stored in the document storage unit 3. The extracted document section describes document information corresponding to a single speaker's utterance. The document section extraction means 6 extracts the document section by using, for example, a method of using the document format information, a method of using the description about the speaker entered in the document, or a method of using metadata in the structured document. Do.

  The section correspondence relationship deriving unit 7 derives a correspondence relationship between the content section extracted by the content section extracting unit 5 and the document section extracted by the document section extracting unit 6 and outputs it to the output unit 8. The output unit 8 displays, outputs, and stores the correspondence relationship on a display device, a printer, an internal storage device, an external storage device, or the like.

  When the document association device 10 is realized by a computer, the content section extraction means 5, the document section extraction means 6 and the section correspondence relationship derivation means 7 are a computer processing unit (example: CPU) and each means 5, This can be realized with a program for realizing the functions 6 and 7.

  FIG. 2 is a block diagram showing an example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. The content section extracting unit 5 includes an audio dividing unit 21, an audio feature amount deriving unit 22, a primary storage unit 23, an audio feature amount matching unit 24, and an output unit 25. The audio dividing unit 21 extracts a silent section from the content read from the content storage unit 3 and performs first audio division. The speech feature amount deriving unit 22 derives a speech feature amount for the first speech section obtained by the first division. The primary storage unit 23 stores the start time of the first speech section and the speech feature amount. The audio feature amount matching unit 24 compares the audio feature amount derived by the audio feature amount deriving unit 22 with the audio feature amount stored in the primary storage unit 23. The output unit 25 outputs the processing result of the voice feature amount matching unit 24 to the section correspondence relationship deriving unit 7.

  An example of the operation of the content section extraction means 5 in the embodiment of the document association method of the present invention will be described. FIG. 3 is a flowchart showing an example of the operation of the content section extraction means 5 in the embodiment of the document association method of the present invention. FIG. 3 is shown in FIG. Here, a case where the content is a video including audio and audio analysis is used for content section extraction will be described as an example.

  The audio dividing unit 21 performs first audio division (step S101). That is, the audio dividing unit 21 extracts a silent interval of the input video as the first audio division, and detects an audio interval between the two silent intervals. The silent section is extracted by measuring the audio track of the input video or the audio power of the input audio. The voice feature quantity deriving unit 22 derives a voice feature quantity for the first voice section obtained by the first division of the voice (step S102). Examples of the voice feature amount include an average fundamental frequency, an average utterance time length, and an average voice power of the voice in the section. When the speech feature value deriving unit 22 derives the speech feature value, the primary storage unit 23 determines whether or not the start time of the first speech segment and the speech feature value are stored (step S103). . When the start time and the voice feature amount of the first voice section are not stored, the primary storage unit 23 stores the start time and the voice feature amount of the first voice section (step S104).

When the start time and the voice feature amount of the first voice section are already stored, the voice feature amount matching unit 24 stores the new voice feature amount derived by the voice feature amount deriving unit 22 and the primary storage unit 23. Comparison with the stored voice feature amount is performed (step S105). If the voice feature values in both sections are smaller than (similar to) a preset threshold value, the voice feature value matching unit 24 determines that the utterance by the same person continues (step S106: YES). If the voice data has not ended (step S109: NO), the voice division unit 21 extracts voice information up to the next silent section (step S101).
If the voice feature values in both sections are different (step S106: NO), the voice feature value matching unit 24 determines that the voice speaker has changed. The output unit 25 outputs a section between the start time stored in the primary storage unit 23 and the start time of the current speech section as a single speaker's speech section (step S107). That is, an utterance section of a single speaker is detected by sound feature change point analysis. At the same time, the primary storage unit 23 updates the voice feature amount and the start time to the newly obtained one (step S108). If the voice data has not ended (step S109: NO), the voice dividing unit 21 continues to extract a silent section of the next voice (step S101).

  The above processing is continued until the voice data is finished. Note that here, as the voice feature value, in order to obtain a change in prosodic information features such as voice pitch, voice length, and voice volume (an example of a change in voice feature value), the average fundamental frequency The average speech duration and average voice power are used. However, another scale representing prosodic information may be used. Moreover, you may utilize the feature-value of conversation form, such as a wording and a mustache. In that case, a change in the characteristics of at least one prosodic information may be used.

  Further, here, the content section extraction means 5 identifies the speaker section by detecting the change point of the speaker based on the similarity of the speech feature amount in the speech section. By detecting the point where the speaker has changed rather than identifying the speaker, it is possible to detect the speaker section with higher accuracy than speaker identification or speech recognition. Of course, the content section extraction means 5 may identify the speaker from the voice feature amount at each time and extract the speaker section from the speaker identification result.

  FIG. 4 is a block diagram showing another example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. The content section extracting unit 5 includes a scene dividing unit 31, a person extracting / person feature amount deriving unit 32, a primary storage unit 33, a person feature amount matching unit 34, and an output unit 35. The scene division unit 31 extracts a first video section composed of continuous frames by detecting a scene change from the content read from the content storage unit 3. The person extraction and person feature quantity deriving unit 32 derives a person feature quantity for the first video section. The primary storage unit 33 stores the start time of the first video section and the person feature amount. The person feature amount matching unit 34 compares the person feature amount derived by the person feature amount deriving unit 32 with the person feature amount stored in the person feature amount and start time storage unit 33. The output unit 35 outputs the processing result of the person feature amount matching unit 34 to the section correspondence relationship deriving unit 7.

  Another example of the operation of the content section extraction means 5 in the embodiment of the document association method of the present invention will be described. FIG. 5 is a flowchart showing another example of the operation of the content section extraction means 5 in the embodiment of the document association method of the present invention. Here, a case is assumed in which video information is assumed as an input, and a speaker section is derived on the assumption that a talking speaker is reflected in the video.

  The scene segmentation unit 31 measures a difference between frames of the input video to detect a portion where the video information has changed greatly, and extracts a first video section composed of visually continuous frames based on the detection result. (Step S201). The person extraction and person feature quantity derivation unit 32 extracts a person area shown in the video, performs video processing on the person area, and derives a person feature quantity (step S202). As a method for extracting a person area, when the moving object in the video is only a person, an area where the difference value from the previous frame, which is a method widely used in the field of monitoring as a background difference method, is a specified value or more is used. A method employed as a person area is exemplified. Examples of the feature amount of the person include a face feature amount described in detail such as the shape of the face, and a low-order visual feature amount describing the color distribution, pattern, and boundary shape of the entire person. By using color distribution and patterns, it is possible to take into account the characteristics of the clothes worn by the user (visual characteristics of the clothes of the person). Is fully applicable.

  Whether or not the person feature and start time storage unit 33 stores the start time and person feature of the first video section when the person extraction and person feature deriving unit 32 derives the person feature. Is determined (step S203). When the start time and person feature amount of the first video section are not stored (step S203: NO), the start time and person feature amount of the first video section are stored (step S204). That is, the first video section is detected by analyzing the change point of the visual feature in the video. When the start time of the first video section and the person feature amount are already stored (step S203: YES), the person feature amount matching unit 34 extracts the new person feature derived by the person extraction and person feature amount deriving unit 32. The amount is compared with the person feature quantity stored in the person feature quantity and start time storage unit 33 (step S205). Then, the person feature amount matching unit 34 determines that the utterance by the same person is continued when the person feature amounts of both sections are more similar than a preset threshold value (step S206: YES). When the video data has not ended (step S209: NO), the scene division unit 31 extracts a portion where the next video information has changed significantly (step S201).

  The person feature amount matching unit 34 determines that the speaker in the video has changed when the person feature amounts in the two sections are different (step S206: NO). The output unit 35 outputs a section between the start time stored in the primary storage unit 33 and the start time of the current video section as an utterance section of a single speaker (step S207). At the same time, the primary storage unit 33 updates the person feature amount and the start time to the newly obtained one (step S208). When the video data has not ended (step S209: NO), the scene division unit 31 extracts a portion where the next video information has changed significantly (step S201).

  The above processing is continued until the video data is finished. Examples of the video feature amount include low-order feature amounts such as color distribution, shape, and edge histogram, and higher-order feature amounts such as eye category, eye, nose, and mouth arrangement. Further, as the feature amount, an appropriate one may be adopted, or a plurality may be combined. If the assumption that the person does not move greatly is introduced, it is possible to use the visual feature amount including the background information without extracting the person region.

  FIG. 6 is a block diagram showing still another example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 6 shows content section extraction means 5 that performs both section extraction related to audio and section extraction related to video. The speech section extraction unit 81 includes, for example, the speech division unit 21, the speech feature amount derivation unit 22, the primary storage unit 23, the speech feature amount matching unit 24, and the output unit 25 illustrated in FIG. The video section extraction unit 82 includes, for example, the scene division unit 31, the person extraction / person feature amount derivation unit 32, the primary storage unit 33, the person feature amount matching unit 34 and the output 35 shown in FIG. 4. The audio / video segment extraction unit (audio / video segment integration means) 83 determines a content segment from the output of the audio segment extraction unit 81 and the output of the video segment extraction unit 82. For example, the audio / video segment extraction unit 83 determines the content segment by adopting only the time point when both the output of the audio segment extraction unit 81 and the output of the video segment extraction unit 82 indicate that the speaker has changed.

  FIG. 7 is a flowchart showing still another example of the operation of the content section extracting means 5 in the embodiment of the document association apparatus of the present invention. The audio segment extraction unit 81 divides the input video into a plurality of audio segments based on the audio of the input video (step S121). For example, the operation shown in FIG. 3 is executed. On the other hand, the video segment extraction unit 82 divides the input video into a plurality of video segments based on the video of the input video (step S122). For example, the operation shown in FIG. 5 is executed. However, step S121 and step S122 may be performed simultaneously, or step S122 may be performed first. Next, the audio / video segment extraction unit (audio / video segment integration means) 83 determines a content segment based on the output of the audio segment extraction unit 81 and the output of the video segment extraction unit 82 (step S123). For example, the audio / video segment extraction unit 83 determines the content segment by adopting only the time point when both the output of the audio segment extraction unit 81 and the output of the video segment extraction unit 82 indicate that the speaker has changed.

  FIG. 8 is a block diagram showing another example of the configuration of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention. FIG. 8 shows content section extraction means 5 that extracts a single speaker section of content using both audio analysis and video analysis.

  The scene dividing unit 91 analyzes the feature amount of the content and divides it into scenes. The scene division unit 91 may use audio feature amounts like the audio division unit 21 shown in FIG. 2, or visual features like the person extraction and person feature quantity derivation unit 32 shown in FIG. An amount may be used. Further, the sum of the voice feature quantity and the person feature quantity may be taken. That is, in order to derive the point of time when the speaker changes, it is also possible to perform a change point analysis of visual features in a video and a change point analysis of sound features in speech to integrate both results. The audio feature amount deriving unit 92 derives the audio feature amount of the extracted scene. The visual feature amount deriving unit 93 derives the visual feature amount of the extracted scene. When the extracted voice feature quantity and visual feature quantity are not accumulated, the primary storage unit 94 stores the voice feature quantity and visual feature quantity and the start time thereof. If the audio feature quantity and the visual feature quantity are already stored, the audio feature quantity matching unit 95 and the audio feature quantity input from the audio feature quantity deriving unit 92 and the audio feature quantity stored in the primary storage unit 94 are stored. Compare with. Similarly, the visual feature amount matching unit 96 compares the visual feature amount input from the visual feature amount deriving unit 93 with the visual feature amount stored in the primary storage unit 94.

  When the difference between the speech feature amount input from the speech feature amount deriving unit 92 and the speech feature amount stored in the primary storage unit 94 is larger than a predetermined threshold, or input from the visual feature amount deriving unit 93 When the difference between the visual feature value and the visual feature value stored in the primary storage unit 94 is larger than a predetermined threshold, the voice feature value and the visual feature value stored in the primary storage unit 94 are cleared. The current time and start time are sent to the output unit 97. The output unit 97 outputs them to the section correspondence relationship deriving means 7. It should be noted that the difference between the speech feature amount input from the speech feature amount deriving unit 92 and the speech feature amount stored in the primary storage unit 94 is greater than a predetermined threshold value and is input from the visual feature amount deriving unit 93. When the difference between the visual feature value and the visual feature value stored in the primary storage unit 94 is larger than a predetermined threshold value, the current time and the start time may be sent to the output unit 97. Good.

  FIG. 9 is a flowchart showing another example of the operation of the content section extraction means 5 in the embodiment of the document association apparatus of the present invention.

  The scene division unit 91 analyzes the feature amount of the content and divides it into scenes (step S141). The scene division unit 91 may use audio feature amounts like the audio division unit 21 shown in FIG. 2, or visual features like the person extraction and person feature quantity derivation unit 32 shown in FIG. An amount may be used. Further, the sum of the voice feature quantity and the person feature quantity may be taken. That is, in order to derive the point of time when the speaker changes, it is also possible to perform a change point analysis of visual features in a video and a change point analysis of sound features in speech to integrate both results. The audio feature amount deriving unit 92 derives an audio feature amount of the extracted scene (step S142). The visual feature amount deriving unit 93 derives the visual feature amount of the extracted scene (step S143). However, step S142 and step S143 may be performed simultaneously, or step S143 may be performed in bloom. The primary storage unit 94 determines whether or not the extracted audio feature quantity and visual feature quantity are accumulated (step S144). When the extracted audio feature quantity and visual feature quantity are not accumulated (step S144: NO), the primary storage unit 94 stores the audio feature quantity and visual feature quantity and the start time thereof (step S145). .

  When the audio feature quantity and the visual feature quantity are already stored (step S144: YES), the audio feature quantity matching unit 95 stores the audio feature quantity input from the audio feature quantity deriving unit 92 and the primary storage unit 94. Comparison with the voice feature value. Similarly, the visual feature amount matching unit 96 compares the visual feature amount input from the visual feature amount deriving unit 93 with the visual feature amount stored in the primary storage unit 94 (step S146).

  When the difference between the speech feature amount input from the speech feature amount deriving unit 92 and the speech feature amount stored in the primary storage unit 94 is smaller (similar) than a predetermined threshold, and the visual feature amount When the difference between the visual feature value input from the derivation unit 93 and the visual feature value stored in the primary storage unit 94 is smaller (similar) than a predetermined threshold value, the audio feature value derivation unit 92 and The visual feature quantity deriving unit 93 determines that the utterance by the same person is continuing (step S147: YES). If the data has not ended (step S150: NO), the scene division unit 91 continues the scene division (step S141).

  When the difference between the speech feature amount input from the speech feature amount deriving unit 92 and the speech feature amount stored in the primary storage unit 94 is larger than a predetermined threshold, or input from the visual feature amount deriving unit 93 When the difference between the visual feature value and the visual feature value stored in the primary storage unit 94 is larger than a predetermined threshold value, the audio feature value deriving unit 92 or the visual feature value deriving unit 93 is the same. It is determined that the utterance by the person has ended (step S147: NO). The primary storage unit 94 clears the stored audio feature quantity and visual feature quantity, and sends the current time and start time to the output unit 97 (step S148). The output unit 97 outputs them to the section correspondence relationship deriving means 7 (step S149).

It should be noted that if the difference between the speech feature amount input from the speech feature amount deriving unit 92 and the speech feature amount stored in the primary storage unit 94 is larger than a predetermined threshold, and the visual feature amount deriving unit 93 When the difference between the input visual feature value and the visual feature value stored in the primary storage unit 94 is larger than a predetermined threshold value, it is determined that the utterance by the same person continues and the current time And the start time may be sent to the output unit 97.
In that case, when the difference between the speech feature amount input from the speech feature amount deriving unit 92 and the speech feature amount stored in the primary storage unit 94 is smaller than a predetermined threshold, or the visual feature amount deriving unit 93. When the difference between the visual feature amount input from the visual feature amount and the visual feature amount stored in the primary storage unit 94 is smaller than a predetermined threshold value, it is determined that the utterance by the same person has ended.

  In this way, speaker segments that could not be distinguished by speech were identified from the video, and speaker segments that were difficult to detect from the video because of similar visual features such as face or clothes were identified. It can be extracted by voice feature. That is, it becomes possible to detect the content section with high accuracy.

  The document section extraction means 6 shown in FIG. 1 extracts sections (document sections) corresponding to the individual speakers in the document from the document information stored in the document storage means 4. In the extracted document section, document information corresponding to a single speaker's utterance is described. To extract a document section corresponding to a speaker from document information, for example, a method using document format information, a method using a description about a speaker entered in a document, or metadata in a structured document is used. There is a way.

  FIG. 10 is a flowchart showing an example of the operation of the document section extraction means 6 in the embodiment of the document association apparatus of the present invention. The document section extraction unit 6 extracts information indicating a document break (hereinafter, “document break information”) from the document information stored in the document storage unit 4 (step S161). The document delimiter information is exemplified by a line feed (blank line) in a document, a character font difference, a character color difference, a character layout, a description of a speaker's name, and the like. Next, the document section extraction means 6 selects an optimal document section extraction method based on the document delimiter information (step S162). The correspondence (table) between the document segmentation information and the document segment extraction method is stored in a storage unit (not shown). The document segment extraction method corresponding to the speaker from the document information includes, for example, a method using the document format information, a method using the description about the speaker entered in the document, and using metadata in the structured document. There is a way to do it. Then, the document section extracting means 6 extracts a section (document section) corresponding to each speaker in the document. In the extracted document section, document information corresponding to a single speaker's utterance is described. However, if the document information is determined in advance, steps S161 and S162 may be omitted, and the document segment extraction method corresponding to the document information may be immediately executed.

Hereinafter, a specific example of the document segment extraction method executed by the document segment extraction means 6 will be described.
11A to 11D are diagrams illustrating an example of a method of using document format information in the embodiment of the document association method of the present invention. In the example illustrated in FIG. 11A, a blank line is inserted for the speech between the speakers. Therefore, the document section extraction means 6 can extract a document section based on a blank line. In the example shown in FIG. 11B, a document in a conversation is illustrated. The host's remarks are displayed in italics. Therefore, the document section extraction means 6 can extract the document section by discriminating between the guest's speech content and the host's speech content. In the example shown in FIG. 11C, the color is different for each speaker. Often used to distinguish between multiple speakers. Therefore, the document section extraction means 6 can extract the document section using the color information. In the example illustrated in FIG. 11D, the description locations are arranged for each speaker. In this way, when the description locations are arranged for each speaker, the document segment extraction means 6 extracts a segment estimated as a single speaker even if the name of the speaker is not directly entered. be able to. Note that the sections extracted here are only candidates and are preferably divided by sections of a single speaker, but may not be strictly grouped with a single speaker. In the method described with reference to FIGS. 11A to 11D, an example of document structure analysis is performed.

  12A to 12C are diagrams showing another example of the method using the document format information in the embodiment of the document association method of the present invention. FIG. 12A to FIG. 12C show a method of extracting a document section using a description about a speaker entered in a document. In the example shown in FIG. 12A, the speaker is entered in the form of “name:” before the utterance. The document section extraction means 6 can extract a document section based on “name:”. In the example shown in FIG. 12B, expressions such as “Quest” and “Answer” are used instead of names. The document section extraction means 6 can extract a document section based on “Question” and “Answer”. In the example shown in FIG. 12C, the name of the speaker is displayed in a separate column, which is widely used in drama scripts and minutes. If such information is used, the document segment extraction means 6 can easily extract information on the speaker and the speaker segment from the document. Note that an example of document structure analysis is also performed in the method described with reference to FIGS. 12A to 12C.

  FIG. 13 is a diagram showing still another example of a method of using document format information in the embodiment of the document association method of the present invention. FIG. 13 shows a method for extracting a document section using a tag in a structured document. The document section extraction means 6 can extract a document section by using, for example, a “Speaker” tag. In addition to the method of extracting a document section from the document illustrated in FIGS. 11A to 13, document section extraction using document format information and a description about a speaker is possible. Further, the document section extraction means 6 can extract the speaker section with higher accuracy by combining these methods. Further, the document section extracting means 6 may derive a document section based on a change in conversation features such as a speech or phrase of a conversation-corresponding portion of a description in the document, as in the case of speech. Note that an example of document structure analysis is also performed in the method described with reference to FIG.

  Next, the section correspondence relation deriving means 7 in the embodiment of the document correspondence apparatus of the present invention will be described. FIG. 14 is a block diagram showing an example of the configuration of the section correspondence relationship deriving means 7 in the embodiment of the document association apparatus of the present invention. In the example illustrated in FIG. 14, the section correspondence relationship deriving unit 7 includes a content length normalization unit 41, a document length normalization unit 42, a section matching degree deriving unit (section matching unit) 43, and a section correspondence relationship storage unit 44. And a section integration unit 45 and an output unit 46. The content length normalization unit 41 normalizes the content length in each extracted section. The document length normalization unit 42 normalizes the length of each document section. The section matching degree deriving unit (section matching means) 43 derives the correspondence between the content section and the document section. The section correspondence storage unit 44 stores the correspondence for each section. The section integration unit 45 integrates adjacent sections and associates content and documents on a one-to-one basis. The output unit 46 outputs the correspondence relationship.

  Next, the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention will be described. FIG. 15 is a flowchart showing an example of the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention. 16A and 16B are diagrams illustrating the correspondence between content information and document information in the correspondence derivation method. FIG. 17 is a diagram for explaining normalization in the correspondence derivation method. In the example shown in FIG. 16, for simplification of explanation, the speaker sections extracted by the content section extraction unit 5 are six sections ([a] − [f]), and the document extracted by the document section extraction unit 6. It is assumed that the section is 7 sections ([1]-[7]).

  The content length normalization unit 41 normalizes the content length in each extracted section (step S301). At the time of normalization, when the content includes audio as shown in FIG. 17A, first, a silent part in each section is extracted. Next, the extracted silent part is removed from each section. Then, normalization is performed so that the length of each section is proportional to the length of the voice part, and the total sum is 1.0. This state is shown in FIG. It is assumed that the content information shown in FIGS. 16A (a) and 17 (a) includes a silent part. Also, as shown in FIG. 17C, normalization may be performed so as to be proportional to the mere section length without removing the silent part. When the content does not include audio, person detection is performed from the video information, and when the content is not included, the length of each section is proportional to the length of the audio portion, and the sum is 1.0. You may normalize to. You may normalize so that it may be proportional to a mere section length, without removing the section which does not contain a person.

  The document length normalization unit 42 normalizes the length of each document section (step S302). For example, the length of each section is set to a length proportional to the document amount (or character amount) included in each section. An example of the result of normalizing both is shown in FIG. 13A. 16A (a) shows content information, and FIG. 16A (b) shows document information.

  The section matching degree deriving unit 43 derives an individual correspondence between the content section and the document section (step S303). For example, the overlap on the normalization axis is examined, and it is assumed that there is a correspondence with the most overlapped area. In the example shown in FIG. 16A, the correspondence relationship is [1] → [a], [2] → [a], [3] → [b], [4] → [c], considering document information. [5] → [d], [6] → [f], [7] → [f]. Considering content information, [a] → [2], [b] → [3], [c] → [4], [d] → [5], [e] → [5], [f] → [7] The section correspondence storage unit 44 stores the correspondence for each section derived by the section matching degree deriving unit 43.

  The section integration unit 45 determines whether or not the content and the document completely correspond to each other (Step S304). When the content and the document do not completely correspond one-to-one (step S304: NO), the section integration unit 45 determines the content and the document based on the correspondence for each section stored in the section correspondence storage unit 44. The adjacent sections are integrated so that the content and the document are in a one-to-one correspondence (steps S304 and S305). For example, the process of integrating adjacent sections corresponding to the same section (example: [1] → [a], [2] → [a] when [1] and [2] are integrated) is repeated) Thus, a one-to-one correspondence between the content and the document can be obtained. When the content and the document completely correspond one-to-one (step S304: YES), the output unit 46 regards the section after integration by the section integration unit 45 as one section and outputs the correspondence (step S306). ).

  In the example shown in FIG. 16A, the above processing results in [[1] [2] ⇔ [a]], [[3] ⇔ [b]], [[4] ⇔ [c] as shown in FIG. 13B. ], [[5] ⇔ [d] [e]], [[6] [7] ⇔ [f]] and the sections can be extracted. As described above, the section correspondence relationship deriving means 7 performs the association by comparing the section length of the extracted content section with the document amount of the extracted document section.

  The section correspondence relationship deriving unit 7 can also derive the correspondence relationship by introducing the certainty of content change. In other words, in addition to the derived section information, the certainty of change point extraction used for section extraction is input as a score from the content section extraction means 5, and the correspondence is derived using the certainty of change point extraction. To do. For example, in an area where the certainty of change is high, the section integration unit 45 integrates one section with another section instead of performing the integration process on both sections having a high score of change confidence. 18A and 18B are diagrams showing the correspondence between content information and document information in the correspondence derivation method. That is, in the example shown in FIG. 15A, when the certainty of change of [d] → [e] is 0.90 (high) and the certainty of [e] → [f] is 0.40 (low), [E] having a short length is integrated with [f] to derive a correspondence. As a result, as shown in FIG. 15B, a correspondence that reflects the certainty factor can be derived.

  In addition, the same processing can be performed when the certainty factor at the time of document segment extraction is used instead of the certainty factor of the content segment, or when the certainty factor is used in both the content segment and the document segment. is there.

  FIG. 19 is a block diagram showing another example of the configuration of the section correspondence relationship deriving means 7 in the embodiment of the document correspondence apparatus of the present invention. The section correspondence relationship deriving unit 7 includes a speaker information storage unit 51, a speaker identifying unit 52, a document speaker information extracting unit 53, and a section matching degree deriving unit 54. The speaker information storage unit 51 stores the correspondence between the feature quantity for specifying a person and the person. The speaker identification unit 52 identifies a speaker. The document speaker information extraction unit 53 extracts information about the speaker from the document. The section matching degree deriving unit 54 performs section matching based on the speaker information.

  The speaker information storage unit 51 records in advance the correspondence between a feature quantity (including a voice feature quantity or a visual feature quantity) for specifying a person and the person. The feature amount is set in advance for person identification. For example, when using speech feature values, speaker-specific feature values that differ for each speaker, such as pitches and pitches for specific phonemes or words, are used. Information such as wording and moustache may be used. When using visual features, the shape, positional relationship, and the like of eyes, nose, and mouth are used as features of the speaker's face. As the feature amount, a known feature amount used as a face recognition technique or a speaker identification technique can be used.

  The speaker identification unit 52 inputs the content section information and the feature amount included in the section from the content section extraction means 5 and compares them with the feature amount stored in the speaker information storage unit 51. Identify speakers in one or more segments. As described above, the speaker identifying unit 52 as the feature amount matching identifying unit includes the feature amount stored in the speaker information storage unit 51 and the feature extracted by the content feature amount extracting unit (specifically, the content section extracting unit 5). The speaker is identified by comparing with the quantity. For example, the speaker identification unit 52 extracts a person in the speaker information storage unit 51 having the closest input feature amount. When characters are limited in advance in a conference or a TV program, identification may be performed in consideration of the restriction information, or all candidate speakers may be listed. The document speaker information extraction unit 53 extracts speaker information (speaker information) from the document by specifying speakers in one or more document sections. The section matching degree deriving unit 54 performs section matching based on the speaker information. That is, the speaker section is associated with the document section.

  Next, another correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention will be described. FIG. 20 is a flowchart showing another example of the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention. 21 and 22 are diagrams showing the correspondence between content information and document information in the correspondence derivation method. This example is effective when the speaker information is described in the document and can be extracted as shown in FIGS.

  The speaker identification unit 52 compares the content section information inputted from the content section extraction means 5 and the feature amount included in the section with the feature amount stored in the speaker information storage unit 51. Thus, the speaker (speaker section) in one or a plurality of sections is specified (step S321). On the other hand, the document speaker information extracting unit 53 extracts speaker information (speaker information) from the document by specifying speakers in one or more document sections (step S322). However, step S321 and step S322 may be performed simultaneously, or step S322 may be performed first. Next, the section matching degree deriving unit 54 performs section matching based on the speaker information. That is, the speaker section and the document section are associated with each other (step S323). It works in this way.

  In an example of the regular heavy rain processing of the section by the section matching unit 54 shown in FIG. 21 ((a) content information, (b) document information), the speaker identification unit 52 performs speaker information based on the content information (: content section). Correspondence between sections is taken according to person identification information that is a result of specifying a speaker by using a feature amount stored in the storage unit 51. Regarding the correspondence between the sections, a method of dynamic programming matching (DP matching) may be introduced. When “Tanaka” is not extracted as illustrated in FIG. 21 because the speaker identification accuracy based on the content information is low, it is possible to take action by skipping “Tanaka”.

  FIG. 22 ((a) content information, (b) document information) is for explaining an example of section matching processing by the section matching degree deriving unit 54 when the speaker identifying unit 52 extracts a plurality of persons as candidates. It is explanatory drawing. In this case, the area [f] can be associated with the section [7] of the document information by the person information based on the document information. It is assumed that “Takagi” and “Yamashita” do not appear in the document. The section [a] is a section of “Yamamoto” or “Tanaka”, but since both names appear in the document information, they are associated with [1] and [2].

  FIG. 23 is a block diagram showing another example of the configuration of the section correspondence derivation means 7 in the embodiment of the document association apparatus of the present invention. The section correspondence relationship deriving unit 7 includes a speech recognition unit 61 that performs speech recognition and generates candidate text for the input speech, and a candidate text document correspondence unit 62 that associates the candidate text with the document in the document storage unit 4. Including.

  FIG. 24 is a block diagram illustrating an example of the configuration of the candidate text document corresponding unit 62. The candidate text document correspondence unit 62 includes a candidate text word extraction unit 71, a document segment word extraction unit 72, a candidate text / document segment correspondence unit 74, and a candidate text / document segment word similarity calculation unit 73. . The candidate text word extraction unit 71 extracts one or more words from the candidate text in the section. The word extraction unit 72 in the document section extracts one or a plurality of words in each section. The candidate text / document section correspondence unit 74 associates each section. The candidate text / document section word similarity calculation unit 73 calculates a section distance.

Next, another correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention will be described. FIG. 25 is a flowchart showing another example of the correspondence derivation method executed by the section correspondence derivation means 7 in the embodiment of the document association method of the present invention. 26 and 27 are diagrams showing the correspondence between content information and document information in the correspondence derivation method. Assume that the content includes audio information.
The voice recognition unit 61 receives information about the content section from the content section extraction means 5. Further, content information is input from the content storage means 3. Then, voice information is extracted from the content information, voice recognition is performed, and candidate text for the input voice is generated (step S341). There are various methods for speech recognition, but any method in this embodiment such as a recognition method using phonemes, a speech recognition method using a direct word template, a template conversion according to the speaker, etc. May be used.

  The candidate text document correspondence unit 62 receives the candidate text of each section of the content from the speech recognition unit 61 and associates the candidate text with the document in the document storage unit 4.

  The candidate text document correspondence unit 62 compares the words in the candidate text with the words in the document section. Then, the content section including the matched word or the similar word is associated with the document section. Specifically, the word extraction unit 71 in candidate text extracts one or more words used in each content section from the candidate text in the section (step S342). The document segment word extraction unit 72 extracts one or a plurality of words in each document segment (step S343). Note that step S342 and step S343 may be performed simultaneously, or step S343 may be performed first. Next, the candidate text / document section word similarity calculation unit 73 calculates a section distance for determining the similarity between the word in the content section and the word in the document section (step S344). The candidate text / document section correspondence unit 74 compares the extracted word sets with each other based on the intra-section distance, associates the content section with the document section, and outputs the result (step S345).

  FIG. 26 shows an example of the correspondence between the candidate text and the document in the document storage unit 4 by the candidate text document section correspondence unit 74. (A) is a content section, (b) is a start time of the content section, (c) is a candidate text word, (d) is a word in a document section, (e) is a document section, and (f) is a document. In the example shown in FIG. 26, in each document section, as words (basic words characterizing the contents of the document section) that are important in the document section, (information communication, speech recognition, semantic information,...), (Security, Video camera, moving object, ...), (experiment, ...), (research, ...) are extracted. Each audio video section, that is, a content section (13:41, 15:41), (15:41, 16:50), (16:50, 20:15), (20:15, 21:13),... From these, words such as (voice recognition, semantic information,...), (Information communication, semantic information,...), (Security,...), (Research,...) Are extracted. . Such words may be obtained by simply extracting nouns from the document, or may be extracted by registering important words in the dictionary in advance and matching the words in the dictionary. The importance may be determined by word usage frequency analysis.

  FIG. 27 shows an example of the correspondence between the candidate text and the document in the document storage unit 4 by the candidate text document section correspondence unit 74. (A) shows the content section, (b) shows the time of the content section, (c) shows the document section, (d) shows the document, and (e) shows the correspondence table. The candidate text document section correspondence unit 74 can derive the correspondence relation of each section as illustrated in the correspondence relation table in FIG. 27E by measuring the similarity (duplication degree) of the word string. . In addition, as illustrated in FIG. 26, when it is not possible to take a correspondence, it may be determined that “a correspondence cannot be taken”. Further, a dynamic programming matching (DP matching) technique may be used for deriving the correspondence between the content section and the document section.

  As described above, the association between the content section and the document section is realized. The association may be realized by a combination of the components of the section correspondence relationship deriving unit 7 (FIGS. 14, 19, and 23).

  The output means 8 shown in FIG. 1 outputs the correspondence between the audio or video derived by the section correspondence deriving means 7 and the document section. As an example of the output form, as shown in FIG. 27 (e), there is a correspondence table in which the time in the content is added to the head of the document section. In addition, any output form may be used as long as it represents the correspondence between the content time information and the document section.

  The present invention provides an information presentation device that synchronously displays content and document information by automatically associating the content with document information, a multimedia display device that searches for or cues the relevant part of the content with text information, It can be applied to uses such as a multimedia search device. Further, the present invention can be applied to applications such as an assembly video browsing device that confirms actual content while referring to minutes, a lecture support system that refers to lecture materials and lecture contents, and an education support system.

Claims (51)

A document association method using a document association apparatus comprising a content section extraction unit, a document section extraction unit, and a section correspondence relationship derivation unit,
(B) The content section extraction unit divides content including at least one of audio information and video information into a plurality of content sections;
(C) the document section extracting unit divides a document describing the content contents into a plurality of document sections;
(D) The section correspondence relationship deriving unit has normalized the first change pattern indicating how the section lengths of the plurality of content sections in the content are changed by dividing the total section length of the plurality of content sections. A normalized first change pattern and a second change pattern indicating how to change the document amount of the plurality of document sections in the document are divided by the total document amount of the plurality of document sections. Based on the comparison with the two change patterns, the plurality of content sections from the positions of the plurality of content sections in the normalized first change pattern and the positions of the plurality of document sections in the normalized second change pattern the correspondence relationship between the plurality of documents sections were immediately Bei and deriving a,
The step (d) includes:
(D1) Two adjacent contents of the plurality of content sections so that the section correspondence relation deriving unit corresponds one-to-one with the number of the plurality of content sections and the number of the plurality of document sections. And a step of executing at least one of integrating the sections into one content section and integrating two adjacent document sections of the plurality of document sections into one document section. ,
The step (d1) includes
(D11) When the section correspondence relation deriving unit integrates the two content sections into the one content section, the section correspondence relationship deriving unit performs the extraction of a change point as a partition for dividing the plurality of content sections. In the case where the two content sections having relatively low certainty of change point extraction in the change point analysis are selected and the two document sections are integrated into one document section, the plurality of document sections are A document association method including a step of selecting the two document sections having relatively low certainty of change point extraction in change point analysis performed to extract change points as divisions to be divided . The document association method according to claim 1,
The step (d) includes:
(D2) the section correspondence relation deriving unit identifying each speaker of the plurality of content sections in the normalized first pattern based on a feature amount that identifies a person included in the content;
(D3) The speaker of each of the plurality of document sections in the normalized second pattern based on speaker information as information regarding a speaker that identifies the person included in the document, by the section correspondence relationship deriving unit Identifying steps,
(D4) Based on the speaker specified by the normalized first pattern and the normalized second pattern, the section correspondence relationship deriving unit determines the normalized first change pattern and the normalized second change pattern. And a step of associating with a document associating method. The document association method according to claim 1,
The step (d) includes:
(D5) the section correspondence relationship deriving unit deleting a silent section in each of the plurality of content sections;
(D6) The section correspondence relation deriving unit obtains a section length of each of the plurality of content sections from which silent sections are deleted;
(D7) The section correspondence relationship deriving unit includes the step of obtaining the normalized first change pattern based on section lengths of a plurality of content sections from which the silent section is deleted. The document association method according to claim 1,
The step (b)
(B1) the content section extraction unit extracting a feature amount specifying a person included in the content;
(B2) The content section extraction unit estimates a section in which the feature amount is more similar to a preset threshold value as one content section in which one speaker's utterance continues, and adjacent content sections Extracting the plurality of content sections having different speakers. A document association method. The document association method according to claim 1 ,
The step (c) includes:
(C1) the document section extracting unit extracting document delimiter information indicating a document delimiter included in the document;
(C2) The document section extracting unit estimates the section between the document delimiter information as one document section in which a single speaker continues speaking, and the plurality of speakers having different content sections are different. A document matching method comprising a step of extracting a document section of a document. The document association method according to claim 1 ,
The step (d) includes:
(D8) The section correspondence relationship deriving unit sets one of the first change pattern of the plurality of content sections and the second change pattern of the plurality of document sections as a standard pattern and the other as an input pattern. A document association method comprising the step of performing the association by a dynamic programming matching method in pattern recognition. The document association method according to claim 1,
The step (b)
(B3) From the one of the plurality of speakers at the time when the feature amount changes based on the feature amount specifying the person included in the stored content, the plurality of speakers Extracting from the content a point in time when the speaker has changed to the other person;
(B4) The content segment extraction unit includes a step of dividing the content in units of speakers based on a point in time when the speaker changes. In the document matching method according to claim 7 ,
The step (b3)
(B31) The content is the audio information, and the content section extraction unit determines a change point of the speaker's voice as the time when the audio feature amount changes based on the audio feature amount as the feature amount. A document association method including a step of extracting from the audio information. In the document matching method according to claim 7 ,
The step (b3)
(B32) The content is the video information, and the content section extraction unit determines a change point of the video of the speaker as the time when the person feature changes based on the person feature as the feature. A document matching method including a step of extracting from video information. In the document matching method according to any one of claims 1 to 7 ,
The content is audio-video information in which the audio information and the video information are synchronized. In the document matching method according to claim 8 ,
The step (b3)
(B33) The content section extraction unit performs a change point analysis of the voice feature quantity as a sound feature of the voice information, and derives a time point when the speaker changes as a time point when the voice feature quantity changes. Document matching method including In document correspondence method according to claim 9,
The step (b3)
(B34) The content section extraction unit performs a change point analysis of the person feature amount as a visual feature of the video information, and derives a time point when the speaker changes as a time point when the person feature amount changes. Document matching method including steps. In the document matching method according to claim 7 ,
The step (b3)
(B35) The content is audio / video information including the audio information and the video information, and the content section extraction unit analyzes a change point of a person feature amount as the feature amount which is a visual feature of the video information. And a change point analysis of the voice feature quantity as the feature quantity, which is a sound feature of the voice information, and the results of both are integrated to derive a time point when the speaker changes as a time point when the feature quantity changes. Document matching method including the step of performing. In the document matching method according to claim 5 ,
The step (c2) includes:
(C21) The document section extraction unit performs structural analysis of the document based on at least one of a blank line, a font difference, a character color difference, a character layout, and a speaker name as the document delimiter information. A document association method comprising a step of dividing the document into units of speakers. A program that causes a computer to execute the method according to any one of claims 1 to 14 . A storage medium storing the program according to claim 15 readable by a computer. A content segment extraction unit that divides content including at least one of audio information and video information and extracts a plurality of content segments;
A document section extracting unit that extracts a plurality of document sections by dividing a document describing the contents of the content;
A normalized first change pattern that is normalized by dividing a first change pattern indicating how to change a section length of the plurality of content sections in the content by an entire section length of the plurality of content sections; Based on the comparison with the normalized second variation pattern normalized by dividing the second variation pattern of the document amount of the plurality of document segments by the total document amount of the plurality of document segments, the normalized first variation A section correspondence relationship for deriving a correspondence relationship between the plurality of content sections and the plurality of document sections from the positions of the plurality of content sections in the pattern and the positions of the plurality of document sections in the normalized second change pattern and the lead-out portion and ingredients Bei,
The section correspondence relationship deriving unit integrates two adjacent content sections of the plurality of content sections so that the number of the plurality of content sections and the number of the plurality of document sections correspond one-to-one. And executing at least one of making one content section and integrating two adjacent document sections of the plurality of document sections into one document section,
When the two content sections are integrated into the one content section, the section correspondence relation deriving unit performs a change point performed to extract a change point as a break for dividing the plurality of content sections. When the two content sections having a relatively low certainty of change point extraction in the analysis are selected and the two document sections are integrated into one document section, a division for dividing the plurality of document sections A document association apparatus that selects the two document sections having relatively low certainty of change point extraction in change point analysis performed to extract change points as The document association apparatus according to claim 17 , wherein
The section correspondence relation deriving unit identifies each speaker of the plurality of content sections in the normalized first pattern based on a feature amount that identifies a person included in the content, and includes a person included in the document On the basis of speaker information as information relating to a speaker for specifying the speaker, each speaker of the plurality of document sections in the normalized second pattern is specified, and the normalized first pattern and the normalized second pattern A document association apparatus that associates the normalized first change pattern with the normalized second change pattern based on the speaker specified in step (b). The document association apparatus according to claim 17 , wherein
The section correspondence relationship deriving unit deletes a silent section in each of the plurality of content sections, obtains a section length of each of the plurality of content sections from which the silent section has been deleted, and a plurality of the deleted silent sections. A document association apparatus that obtains the normalized first change pattern based on a section length of a content section. In the document matching apparatus according to any one of claims 17 to 19 ,
The content section extraction unit extracts a feature amount that identifies a person included in the content, and one speaker's utterance continues in a section in which the feature amount is similar to a preset threshold value. A document association apparatus that extracts a plurality of content sections that are different from each other by estimating speakers in one content section. In document associating device according to any one of claims 17 to 20,
The document section extraction unit extracts document section information indicating a document section included in the document, and estimates a section between the document section information as one document section in which one speaker's utterance continues. A plurality of document sections having different speakers in adjacent content sections. In the document matching apparatus according to any one of claims 17 to 21 ,
The section correspondence derivation unit recognizes a pattern by using one of the first change pattern of the plurality of content sections and the second change pattern of the plurality of document sections as a standard pattern and the other as an input pattern. A document association apparatus that performs the association by the dynamic programming matching method in FIG. In the document matching apparatus according to any one of claims 17 to 19 ,
The content is the audio information;
The content section extraction unit analyzes a sound feature of the audio information and extracts the plurality of content sections based on a point where the sound feature changes. In the document matching apparatus according to any one of claims 17 to 19 ,
The content is the video information;
The content section extraction unit analyzes a visual feature of the video information and extracts the plurality of content sections based on a point where the visual feature changes. In the document matching apparatus according to any one of claims 17 to 19 ,
The content is audio-video information in which the audio information and the video information are synchronized,
The content section extraction unit integrates the change point of the sound feature in the analysis result of the sound feature of the audio information and the change point of the visual feature in the result of the analysis of the visual feature of the video information. A document association apparatus that extracts the plurality of content sections. The document association apparatus according to claim 25 ,
The content section extractor
Analyzing a sound feature of the sound information, and based on a point where the sound feature changes, a speech section extracting unit that divides the speech information into speaker units and extracts a plurality of speech sections;
Analyzing the visual characteristics of the video information, and based on the points where the visual characteristics change, a video segment extraction unit that divides the video information into speakers and extracts a plurality of video segments;
An audio / video section integration unit that extracts the plurality of content sections based on a plurality of pieces of voice section information about the plurality of voice sections and a plurality of pieces of video section information about the plurality of video sections. In the document matching apparatus according to any one of claims 17 to 19 ,
The content section extraction unit is configured to change another one of the plurality of speakers from one of the plurality of speakers when the feature amount changes based on a feature amount that identifies a person included in the stored content. A document association device that extracts a speaker change point as a time when a speaker changes to one person and extracts the plurality of content sections obtained by dividing the content into units of speakers based on the time when the speaker changes . In document associating device according to claim 27,
The content includes the audio information;
The content section extraction unit determines the speaker change point based on a change in the feature of at least one of the prosodic information among the speech height, the speech speed, and the speech size in the speech information as the feature amount. Document matching device to extract. 28. The document association apparatus according to claim 27 .
The content includes the audio information;
The said content area extraction part extracts the said speaker change point based on the change of the conversation form in the said audio | voice information as the said feature-value Document matching apparatus. 28. The document association apparatus according to claim 27 .
The content includes the video information,
The said content area extraction part extracts the said speaker change point based on the change of the person's visual feature in the said video information as the said feature-value Document matching apparatus. 28. The document association apparatus according to claim 27 .
The content includes the video information,
The said content area extraction part extracts the said speaker change point based on the change of the face feature of the person in the said video information as the said feature-value Document matching apparatus. 28. The document association apparatus according to claim 27 .
The content includes the video information,
The said content area extraction part extracts the said speaker change point based on the change of the visual feature of the clothes of the person in the said video information as the said feature-value Document matching apparatus. The document association apparatus according to any one of claims 17 to 32 ,
The document section extraction unit estimates a section having the same format information based on the format information of the document as one document section in which a single speaker is speaking, and talks between adjacent content sections. A document association device that extracts the plurality of document sections having different persons. The document association apparatus according to any one of claims 17 to 32 ,
The document section extraction unit estimates a section in which the speaker is the same as one document section in which an utterance of one speaker continues based on a description about the speaker entered in the document, and A document associating device that extracts the plurality of document sections having different speakers in matching content sections. The document association apparatus according to any one of claims 17 to 32 ,
The document section extraction unit estimates a section having the same tag information based on tag information of a structured document in the document as one document section in which one speaker's utterance continues and is adjacent to each other. A document association apparatus that extracts the plurality of document sections having different speakers in the content section. The document association apparatus according to any one of claims 17 to 32 ,
The document section extraction unit estimates a section having the same conversation feature based on a change in conversation feature in the document as one document section in which a single speaker is speaking, and is adjacent to a content section. A document association device that extracts the plurality of document sections having different speakers. The document association apparatus according to claim 18 , wherein
The section correspondence relation deriving unit
A content speaker identifying unit that identifies each speaker of the plurality of content sections based on the feature amount;
A document speaker information extraction unit that identifies each speaker of the plurality of document sections based on the speaker information;
A section matching unit that performs matching between the plurality of content sections and the plurality of document sections based on each speaker of the plurality of specified content sections and each speaker of the plurality of document sections. Document association device. 38. The document association apparatus according to claim 37 .
The content speaker identification unit
A content feature amount extraction unit that extracts a feature amount of each of the plurality of content sections;
A speaker information storage unit for storing the feature quantity and the speaker in association with each other;
A document associating device, comprising: a feature quantity matching and identifying unit that identifies the speaker based on a comparison between the stored feature quantity and the extracted feature quantity. In the document matching apparatus according to claim 37 or 38 ,
The content speaker identification unit identifies the speaker based on a feature of at least one prosodic information among the voice pitch, voice length, and voice strength in the voice information as the feature amount. Yes Document matching device. In the document matching apparatus according to claim 37 or 38 ,
The said content speaker identification part specifies the said speaker based on the feature-value which the conversation form in the said audio | voice information as said feature-value represents. Document matching apparatus. In the document matching apparatus according to claim 37 or 38 ,
The content speaker identification unit identifies the speaker based on a visual feature amount of a person in the video information as the feature amount. The document matching apparatus according to claim 41 , wherein
The content speaker identification unit uses a facial feature of a person as a visual feature of the person. In the document matching apparatus according to any one of claims 37 to 42 ,
The said document speaker information extraction part specifies the said speaker based on the description regarding the speaker entered in the said document based on the said speaker information. Document matching apparatus. In the document matching apparatus according to any one of claims 37 to 42 ,
The document speaker information extraction unit specifies a speaker based on metadata of a structured document in the document based on the speaker information. In the document matching apparatus according to any one of claims 37 to 44 ,
The section matching unit associates the plurality of content sections with the plurality of document sections so that a speaker in each of the plurality of content sections matches a speaker in each of the plurality of document sections. Attachment device. The document association apparatus according to claim 45 , wherein
The section matching unit may use one of the first change pattern of the plurality of content sections and the second change pattern of the plurality of document sections as a standard pattern and the other as an input pattern for dynamic in pattern recognition. A document association apparatus that associates the plurality of content sections and the plurality of document sections based on a result of executing programming matching. In the document matching device according to any one of claims 17 to 46 ,
The content includes audio information;
Further comprising a voice recognition unit that extracts utterance contents in the plurality of content sections and outputs utterance text information;
The section correspondence relation deriving unit associates the plurality of content sections with the plurality of document sections based on the similarity between the utterance text information and the document information of the document. 48. The document association apparatus according to claim 47 , wherein
The section correspondence relation deriving unit is a dynamic program for pattern recognition in which one of a word pattern appearing in the utterance text information and a word pattern appearing in the document information is a standard pattern and the other is an input pattern. A document association apparatus that matches the utterance text information and the document information based on a result of matching. In document associating device according to claim 47 or 48, wherein,
The section correspondence relation deriving unit
One or more first basic words used in each of the plurality of content sections in the utterance text information; one or more second basic words used in each of the plurality of document sections; A basic word extraction unit for extracting
A basic word group similarity deriving unit that measures the similarity between the plurality of first basic words and the plurality of second basic words;
A document association apparatus that derives the correspondence relationship based on the similarity. In the document matching apparatus according to claim 47 or 48 ,
The section correspondence relationship deriving unit derives a correspondence relationship by associating the similarity by dynamic programming matching. The document association apparatus according to any one of claims 17 to 50 ,
A content input unit for inputting the content;
A content storage unit for storing the content;
A document input unit for inputting the document information;
A document storage unit for storing the document;
An output unit that outputs information related to the correspondence relationship.

Images