JP3736597B2 - Statement structure information presentation apparatus and statement structure information presentation method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention provides a speech structure information presentation apparatus and speech structure information presentation method suitable for use when searching for a specific section that a user particularly needs to reproduce from conference information including recorded audio information and video information. About.
[0002]
[Prior art]
In a meeting, a lot of information is generated as voice information by conversation. Among these, there is a small amount of information recorded as text information on whiteboards and minutes, and there is a problem that a lot of important information is not recorded or cannot be accurately recalled.
[0003]
In order to solve this problem, there is a conference recording device that records all information generated in a conference. An example of this conference recording device is described in Japanese Patent Application Laid-Open No. 6-343146. Here, all multimedia information such as audio information input from a microphone, video information input from a video camera, text information and graphic information by pen input, etc. is recorded without exception.
[0004]
In such a conference recording device, when trying to remember the content of the conference, how to properly access a necessary place becomes an important issue. However, it is extremely difficult for a participant to attach an index to each meeting scene in real time. In this regard, after the conference is finished, if appropriate indexing is manually performed by a human, effective indexing is possible.
[0005]
However, such an indexing effort is enormous. Furthermore, information required later often changes depending on each person who performs a search or with the passage of time, and it is difficult to perform a sufficient search using a predetermined index. Therefore, a method for automatically providing an effective index from a variety of clue information generated during a conference without disturbing human resources is being studied.
[0006]
Japanese Patent Application Laid-Open No. 6-343146 provides a means for searching for voice and video information using a pen input means using the time when a text or gesture is input as an index. Conference participants often take handwritten notes when important statements occur. From this, it becomes possible to effectively access the important information of the conference by using the time when the handwritten memo is performed as an index.
[0007]
In JP-A-7-182365, corresponding data can be accessed from conference materials. Here, the time zone in which the designated conference material is referenced is presented to the user. When further narrowing down, by excluding other time zones from the search target, it is possible to efficiently access the conference information using the reference material.
[0008]
Japanese Patent Application Laid-Open No. 8-317365 describes that voice data of a conference speaker is graphed in a time series with a length corresponding to the amount of data stored. This makes it possible to visualize the message structure indicating who has made the message and in what order, as a graph.
[0009]
Therefore, even after the conference is over, the conference participant can recall the conference content of the conference in which he / she participated to some extent from the information of the above-mentioned speech structure, and can record important or necessary information in a recording place. It becomes possible to access. The advantage of this technique is that it does not require advanced speech recognition technology or a dictionary, and it can be automatically created from recorded information without the need for explicit instructions from conference participants.
[0010]
"Speaker Segmentation" (Donald G. Kimber, lynn D. Wilcox, Francine R. Chen, and Thomas Moran: "Speaker Segmentation for Browsing Recorded Audio", CHI'95 Proceedings (short paper), pp.212-213) Similarly, a means for visualizing the speech structure is provided. Here, the access structure desired by the accessor can be easily obtained by presenting which part of the displayed speech structure information is in the entire conference. It becomes possible to narrow down.
[0011]
Furthermore, “gIBIS” (Jeff Conklin, KC Burgess Yakemovic “A Process-Oriented Approach to Design Rationale”, HUMAN-COMPUTER INTERACTION, 1991, Vol.6, pp.357-391) is a technology for visualizing the remark structure. is there. Here, in advance, the speaker makes a statement by explicitly declaring what kind of property to speak for which statement. A speech structure is extracted from the attribute information related to the speech and visualized as a speech structure graph. Also, the discussion referencer can access individual statements through the statement structure graph.
[0012]
"Forget-me-not" (Forget-me-not: Close-computed computing that supports human memory ", Cognitive Science Vol.2 as a technology to extract structural units from unstructured information No.1, pp.16-25). Here, human behavior records such as “participating in a meeting”, “meeting a person”, “calling”, and the like are extracted and recorded in a schema according to episode memory, which is human memory. That is, human behavior is structured in memory units called episodes.
[0013]
[Problems to be solved by the invention]
However, in the technique disclosed in Japanese Patent Laid-Open No. 6-343146, there is a problem in that a memo cannot be taken when a conference participant is engaged in discussion. Therefore, such an index that requires active instructions and actions of the conference participants is effective, but there are many leaks. Also, if you want to create a sufficient index, meeting participants will have to take many notes, increasing the burden. Even if a memo is taken, the time of the memo is not necessarily synchronized with the corresponding utterance, and it is difficult to accurately access the desired utterance only from the memo.
[0014]
In addition, in the access by the reference material of JP-A-7-182365, the time zone in which the information related to the reference material exists is extracted, but since it is time series data, the discussion relating to the reference material is performed. I cannot overlook the whole structure.
[0015]
Furthermore, the index using the speech structure of Japanese Patent Laid-Open No. 8-317365 is a method for automatically extracting a sufficient index without imposing a burden on the conference participants as much as possible. Has solved. However, there is a problem in that it is not possible to know which part of the entire conference the displayed speech structure is simply displayed in time series in units of speech from the start time to the end time of the conference.
[0016]
The total meeting time display in “Speaker Segmentation” solves this problem. However, here too, a measure of where in the meeting the desired remark exists must rely on memory.
[0017]
From the above, handwritten memos and meeting materials provide rough information about where the desired information is, but they are not synchronized with the utterance structure and do not provide information to access accurate utterances. . On the other hand, in the case of a speech structure, if a location can be specified, it becomes easy to access a specific speech from the speech structure, but a rough location cannot be specified. That is, there is a possibility that these access means can provide efficient access means by complementing each other.
[0018]
In addition, regarding the speech structure information, there is a component that provides a structure in a unit of speech, but there is no component that provides a partial structure that divides a plurality of speech groups as one lump. On the other hand, gIBIS provides a means for giving a partial structure to an utterance, but it is extremely difficult to make an utterance by determining attributes such as the relation and type of the utterance in advance.
[0019]
Forget-me-not solves the above problem by automatically extracting the structure, but the granularity of the information unit to be extracted is too large to extract as one structure unit from the statement group. It cannot be applied to partial structuring.
[0020]
In view of the above problems, the present invention specifies and presents an appropriate speech structure section for each of a plurality of related speech groups, thereby efficiently accessing a desired location when reproducing conference information. It is an object of the present invention to provide a speech structure information presentation device.
[0021]
[Means for Solving the Problems]
In order to solve the above-described problem, a speech structure information presentation device according to the invention of claim 1 includes:
Material identification information for identifying each of a plurality of meeting materials to be discussed in the meeting, Of the plurality of meeting materials First storage means for storing each presentation time interval information;
Second storage means for storing time information of each speech section of a plurality of conference participants detected from conference audio information and the material identification information of the conference material presented when each speech was made When,
Material discriminating means for discriminating a conference material selected and instructed by the user among the plurality of conference materials;
For the meeting material determined by the material determining means, the first storage means and Said Based on the information stored in the second storage means, in the conference, a time section in which the conference material is presented, and a speech structure indicating an appearance status of the speech section of the conference participant in the time interval in which the conference material is presented; Look Remark structure generating means for generating as awakening information;
It is characterized by providing.
[0022]
According to a second aspect of the present invention, in the statement structure information presentation device according to the first aspect, information on the presentation status of the conference material stored in the first storage unit and the second storage unit A partial utterance structure detecting means for extracting a partial structure of the utterance structure from the mutual relation of the utterance sections of the plurality of conference participants stored therein is provided.
[0023]
Further, the invention according to claim 3 is the speech structure information presentation device according to claim 2, wherein the first speech structure level for extracting all speech sections related to each of the conference materials and the specific presentation of the conference materials A hierarchical structure comprising a second utterance structure level for extracting all utterance sections in a time zone and a third utterance structure level for extracting sections relating to a specific topic detected by the partial utterance structure detection means, It is characterized by comprising speech structure management means for managing the speech structure.
[0024]
According to a fourth aspect of the present invention, in the speech structure information presentation device according to the third aspect, user input information such as memos and markings input by each conference participant at the conference is provided for each conference participant. Third storage means for storing;
Personal information management means for maintaining a relationship between the user input information and the first to third message structure levels managed by the message structure management means, and managing the user input information as a hierarchical structure;
It is characterized by providing.
[0025]
According to a fifth aspect of the present invention, in the speech structure information presentation device according to the fourth aspect, the user input information is designated, and the level of the hierarchical structure is designated, whereby the designated user input information is designated. It is characterized by comprising speech structure section extraction means for specifying and extracting related speech structure sections according to the designated hierarchical level.
[0026]
Further, the invention of claim 6 is the speech structure information presentation device according to claim 4, further comprising display means for displaying the conference material content presented in the conference and the user input information in an overlay manner,
A remark of any one of the first to third remark structure levels according to the reference state of the conference material of the search instruction input by the user on the display means or the selection state of the user input information A speech structure section detecting unit that automatically determines a section and detects only a speech structure section connected to an arbitrary portion of the presentation time section of the conference material according to the speech structure level.
It is characterized by that.
[0027]
The invention according to claim 7 is the speech structure information presentation device according to claim 1, wherein when the length of the speech section stored in the second storage means is longer than a predetermined time, it is detected as an explanatory speech. Explanation speech detection means to
A discussion state determination means for extracting a comment structure of a discussion by deleting an explanatory comment from the comment structure;
Discussion information extraction means for extracting discussion attribute information such as the number of speakers and discussion time from information on the speech structure of the discussion;
Based on the discussion attribute information extracted by the discussion information extraction means, a discussion activation level determination means for calculating a discussion activity level indicating a rising situation of the discussion and determining a discussion activation level;
Means for visually displaying the climax of the discussion on the display screen according to the determination result of the discussion activation level determination means;
It is characterized by providing.
[0028]
[Action]
In the statement structure information presentation device according to the first aspect of the present invention having the above-described configuration, the first storage means stores, for example, the material identification information of the conference material displayed on the display screen and the information of the presentation time interval at the time of the conference. Is remembered.
[0029]
In addition, it is possible to participate in a plurality of conferences by recognizing as a speech a voice signal that has continued for a certain length at a certain volume or higher from input voice information or recorded conference voice information during a conference. Each speech section of the person is detected, and time information of the speech section and the material identification information of the conference material presented when the speech is made are stored in the second storage means.
[0030]
When the user selects and selects one of a plurality of conference materials when reproducing the conference information after the conference is finished, the selected conference material is specified by the material discriminating means, and the specified result is a statement structure diagram. Given to the generating means.
[0031]
The utterance structure diagram generation means includes a second time structure in which the identified conference material is presented, and a utterance structure that indicates the appearance status of all the speech sections of the conference participants in the time interval in which the conference material is presented. Extracted from the storage means and displayed as visualization information on the display screen.
[0032]
Then, when the user designates a desired reproduction location on the message structure diagram screen, the corresponding conference information portion can be reproduced.
[0033]
Thus, according to the invention of claim 1, since the structure of all discussions related to a specific conference material is visualized as a speech structure diagram, it is simply compared with the data displayed in chronological order. This makes it easier to understand the entire discussion about the content. Therefore, when accessing the contents of the conference, it is easy to access the necessary remarks because it is possible to narrow down which remarks should be accessed by grasping the entire context.
[0034]
In the speech structure information presentation device according to the second aspect of the present invention, the partial structure of the speech structure of the conference is extracted from the presentation state of the conference material and the mutual relationship between the speeches of a plurality of conference participants during the conference. For example, if the type of conference is that a specific participant makes a presentation and then discusses according to the content of the presentation, the point of change of the presentation meeting material such as a change of the display page, or more than a certain level by the presenter It detects the break of the discussion by using the attribute of the speech structure, such as the present time of the explanation speech that is a long speech, recognizes the mass of the discussion, and extracts the partial structure of the speech structure.
[0035]
As a result, the partial structure can be extracted from the conference information having no structure, the data can be handled in units of the partial structure, and the search processing efficiency is improved as compared with the case where the partial structure does not exist.
[0036]
In the speech structure information presentation device according to the invention of claim 3, the speech structure is managed at three hierarchical levels. The first speech structure level is a presented conference material unit, and all information related to the speech structure related to an arbitrary conference material is managed at this level. When the same conference material is used multiple times at different times, such as when returning to the previous discussion, it shall be managed as this first speech structure level, including those from different time intervals. become.
[0037]
The second message structure level is a display unit level, and manages all the message structure data in one specific time zone in which a specific conference material is displayed.
[0038]
The third utterance structure level is, for example, a level of discussion units detected by the partial utterance structure detection means of claim 2. For example, explanations and discussions are grouped into one discussion data, with the explanation utterances of the presentation as a separator. Can be extracted and managed.
[0039]
According to the invention of claim 3, when a specific time zone is designated by the user for searching for conference information, it is possible to easily extract a speech structure section at each level and generate a speech structure diagram. Processing efficiency is improved.
[0040]
Further, in the speech structure information presentation device according to the invention of claim 4, each conference participant stores user input information such as memorandum and marking inputted in the conference in the third storage means for each conference participant. And hold. The personal information management means manages the user input information stored in the third storage means in a hierarchical structure in association with the first to third management levels in the statement structure management means in claim 3. .
[0041]
In the case of the invention of claim 4, it becomes possible to easily associate the user input information with the partial structure of the conference material or the speech structure, and the search by the handwritten memo etc. regarding the desired conference information position And the search by the speech structure can be effectively combined, and the access processing efficiency to the speech structure is improved.
[0042]
In the speech structure information presentation device according to the invention of claim 5, when the user designates specific user input information and designates any of the first to third speech structure levels, the speech structure section extraction means The time zone in which the specified user input information is input is specified, and the message structure section at the specified message structure level including the time zone is specified and extracted. The message structure diagram generating means generates and displays a message structure diagram of the specified message structure section.
[0043]
According to the invention of claim 5, when the search target is ambiguous, the user can select the first message structure level that is the level of each conference material, and if the time zone is narrowed down to some extent, When the second speech structure level, which is the level for each display section, and where the location of the speech is narrowed down, the third speech structure level, which is a partial structure level such as a discussion part, is designated, thereby Street search access is possible. That is, the search level can be selected according to the level of clarity of the target to be searched, and efficient narrowing down is possible.
[0044]
In the speech structure information presentation device according to the sixth aspect of the present invention, when the user inputs a conference information search instruction on the display screen on which the conference material content and the user input information are displayed in an overlay manner, the instruction is Whether only the meeting material being displayed is instructed or whether an object is selected is determined. If an object is selected, the display / selection status is determined from a value indicating the display / selection status such as the input time of the selected object.
[0045]
If the object is not selected, the appropriate level is automatically selected, such as the first speech structure level that is a conference material unit, and if the object is selected, the third speech structure level of the partial structure such as discussion The speech structure section corresponding to the conference material displayed at that level and the selected user input information is extracted, and the corresponding speech structure diagram is displayed.
[0046]
According to the sixth aspect of the present invention, even if there is no explicit instruction from the user as the search instruction input, it is possible to draw the intention of the user and present only the speech structure section at an appropriate speech structure level. As a result, the user's search instruction operation is simplified, unnecessary information does not have to be presented, and the load on the user to access the necessary information is reduced.
[0047]
In the speech structure information presentation device of the invention of claim 7, the discussion state determination means deletes the explanatory speech detected by the explanatory speech detection means from the speech sections detected from the conference voice, and Extract speech structure. Then, the discussion information extraction means extracts attributes indicating the status of the discussion such as the presentation time of the conference material, the number of utterances, the number of speakers, etc. in the utterance structure of the discussion, and the activity of the discussion as a composite function of these values Is calculated by the discussion activation level determination means, and the activation level is determined. The activation level is displayed so that the entire material calculated for each conference material can be listed, from which the user can specify a specific conference material to be referred to.
[0048]
Thus, according to the invention of claim 7, when the reference conference material is specified, all the conference information is referred to, such as accessing only the active part of the discussion not only from the content of the material but also from the discussion status of the material. Without having to, you can easily access important discussion parts. Moreover, this information can shorten the time for grasping | ascertaining the content of a meeting for the person who did not participate in a meeting.
[0049]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of a speech structure information presentation device according to the present invention will be described below with reference to the drawings. The speech structure information presentation device according to the embodiment described below is applied to a conference recording / playback device that records conference information for a presentation-type conference and can specify a location to be searched from the conference information. Is the case.
[0050]
Here, the presentation type conference is a type of conference in which a specific conference participant (presenter) makes a presentation and a plurality of conference participants discuss based on the presentation. The speech of the explanation by the presenter is characterized in that the speech section is relatively long compared to the speech of questions including other conference participants thereafter.
[0051]
[Description of Recording Device Part of Conference Recording / Reproducing Device]
FIG. 2 is a diagram for explaining the presentation-type meeting, and is a conceptual diagram showing the state of the meeting in the electronic meeting room.
[0052]
Reference numeral 10 denotes a personal computer for presentation (hereinafter, personal computer is abbreviated as PC), which holds a presentation tool for presenting conference material and a data file of the conference material. At the time of the conference, the display content of the presentation PC 10 is displayed on the large screen 1.
[0053]
The presenter 2 can use a keyboard 3 for character input and a pen input device 4 for instruction input, and perform user input operations such as memo writing and marking on the presentation PC 10. The voice of the presenter 2 is collected by the voice input device 5 such as a microphone and recorded on the conference recording PC 20.
[0054]
Each of the conference participants 6 other than the conference presenter has a local PC 40. The same contents as the large screen 1 are displayed on the local PC 40. Furthermore, the conference participant 6 can input a personal memo from the keyboard 7 and the pen input device 8 to the local PC 40. The input information is recorded and held as a personal information file in each local PC 40. The speech of the conference participant 6 is captured as audio data by the audio input device 9 such as a microphone and recorded on the conference recording PC 20.
[0055]
The conference recording PC 20 integrates the voice data input from the voice input device 5 or the voice input device 9, and records and holds the voice data file as a conference data file. In addition, a process for generating information for an index for accessing the conference information after the conference is performed. The presentation PC 10 and the conference recording PC 20 may be the same PC.
[0056]
FIG. 3 is a block diagram showing a configuration of a part for recording conference information in the conference recording PC 20. The conference recording PC 20 is a device that records the conference information at the time of the conference and performs a process for creating an index for accessing the conference information later.
[0057]
The voice data of the speeches of the presenter 2 and other conference participants 6 from the voice input device 5 and the voice input device 9 are digitized by the A / D conversion device 21. The digitized audio data is taken into the conference recording PC 20.
[0058]
The voice data captured inside is synthesized by the synthesizer 22 with the voice data of a plurality of conference participants including the presenter 2, and is recorded and held in the conference voice data file storage unit 23.
[0059]
Each piece of voice data is supplied to the utterance data extraction unit 24, and the utterance data extraction unit 24 extracts a section of voice data in which a certain volume or more continues for a certain period of time as a utterance section. The utterance data extraction unit 24 extracts a utterance section for each conference participant. As described above, in this example, since a microphone is installed for each conference participant, it is possible to detect which conference participant the audio is from, depending on which microphone is the audio. Yes.
[0060]
The extracted utterance section data is supplied to the utterance structure table generation unit 26 together with the time data of the timer 25, and the utterance structure table which is an index for accessing the speech data of the conference by the utterance structure table generation unit 26. Is generated. As will be described later, the information in the comment structure table indicates the correspondence relationship between the conference participants who have spoken, the time information of the speech section, and the information on the page of the conference material when the speech is detected. is there. The information of the generated message structure table is recorded and held in the message structure file storage unit 27 as a message structure file.
[0061]
As shown in FIG. 4, in the presentation PC 10, the presentation tool 11 is activated during the conference, and the contents of the presentation file 12 are displayed on the display device 13. In the presentation file 12, a plurality of pages of meeting materials are prepared for each page.
[0062]
Here, the presentation tool 11 has a structure in which the display is changed in units of pages. In this embodiment, the conference material is managed in units of pages, and the page number is used as identification information for each conference material. For this reason, when the page is changed during the conference, the presentation tool 11 transmits a display page change notification to the conference recording PC 20.
[0063]
In the conference recording PC 20, when the notification detection unit 28 receives the display page change notification signal, the display information extraction unit 29 extracts information on the conference material page included in the display page change notification signal. The information of the page is recorded and held in the display information file storage unit 30 as a display information file together with the time information of the timer 25. Also, the information on the page of the conference material is sent from the display information extraction unit 29 to the message structure table generation unit 26, and is used as information on which page of the conference material the detected message was performed.
[0064]
FIG. 5 is a diagram for explaining the data structure of the display information file 100. The display information file 100 is a file that holds, as time series data, the transition of the display state of each page of the conference material used in the presentation during the conference. Therefore, this display information file is sometimes called a display information table. Hereinafter, similarly, in this specification, a file in a table format may be referred to as a table.
[0065]
In the display information table 100 of FIG. 5, the field 101 is “display number”, which is a sequential number assigned to display data in page units. A field 102 is a column of “page number”, and records the page number of the conference material displayed in the presentation. A field 103 is a column of “display start time”, and records the time when the display is changed from another page to the current page and the display of the current page is started in the presentation. Similarly, the field 104 is a column of “display end time”, and records the time when the display change occurs to the next page.
[0066]
FIG. 6 is a diagram for explaining the data structure of the message structure table 110. The speech structure table 110 is a data file that holds the speech structure of conference participants in a conference and is used as an index for accessing an audio data file that records the conference.
[0067]
As shown in FIG. 6, in the “speech number” column of the field 111 of this speech structure table 110, identifiers of the detected speech in time order are recorded. In the “participant name” in the field 112, the name of the conference participant detected based on the identifier of the voice input device that extracted the speech is recorded.
[0068]
The speech structure file storage unit 26 also stores a conference participant table, which is a correspondence table between the identifier of the voice input device and the conference participant name. The data structure of the conference participant table 120 is as shown in FIG.
[0069]
That is, the field 121 of the conference participant table 120 is an input device number, which means an audio input device number that is an identifier held by the audio input devices 5 and 9. A field 122 is a conference participant name, and the name of the conference participant assigned to each voice input device is held as text data.
[0070]
When a utterance is detected, first, the voice input device in which the utterance is detected is specified. Then, referring to the conference participant table 120, the corresponding conference participant name is specified. In the field 112 of the statement structure table 110, the participant name specified from the conference participant table 120 is recorded as the participant name.
[0071]
As the “speech start time” in the field 113 of the speech structure table 110, the start time of the extracted speech is recorded as an elapsed time from the recording start time. As the “utterance end time” in the field 114, the end time of the speech is recorded in the same manner. As the “page number” in the field 115, the page number of the conference material page where the corresponding utterance is uttered is recorded.
[0072]
FIG. 6 illustrates the relationship between the speech start time field 112 and the speech end time field 113 of the speech structure table 110 and the audio data file of the conference audio data file storage unit 23. That is, 117 indicates the recording location of the utterance 6 recorded in the conference audio data file storage unit 23. The link 118 points to the speech start point at the recording position of the speech number 6. Similarly, the link 119 points to the speech end point at the recording position of the speech number 6.
[0073]
In the conference recording PC 20 of FIG. 3, the partial structure detection unit 31 is a processing unit that divides the speech structure into some meaningful information units from the features of the speech structure and the display information. In this example, a meaningful information unit is a page unit of a presentation, a display block corresponding to a specific display time during which a certain page was displayed, a presentation explanation and a corresponding question and answer as one unit. Such as discussion blocks.
[0074]
The management file generation unit 32 is also a device for creating a conference information management file (conference information management table) for managing the partial structure of the utterance structure detected by the partial structure detection unit 31 in units of presentation pages. It is. Here, the partial structure information of the utterance is managed in a hierarchical structure of page units, display blocks, and discussion blocks, and the conference information management table is recorded and held in the conference information management file storage unit 33.
[0075]
FIG. 8 is a diagram for explaining the data structure of the conference information management table 130. As the conference information management file, the speech structure data is managed in a three-level hierarchical structure as described above.
[0076]
The highest layer of the conference information management table 130 is in units of pages, and “page number” is recorded in the field 131. In this hierarchy, all message structure data related to the corresponding page is held.
[0077]
The page is composed of a plurality of display blocks which are speech structures of the respective display sections. That is, the display block is a speech structure below the page. In a presentation type meeting, meeting materials are not displayed in order from the first page to the last page. For example, while explaining, a discussion may be inserted in between, a gap may be inserted, another page may be moved, or the original page may be returned.
[0078]
Therefore, the management of only page units cannot manage the time intervals displayed discretely. The display block is for managing discrete display time intervals on the same page, and indicates a interval in which a specific page is continuously displayed in a specific time zone. The “display block number” in the field 132 of the conference information management table 130 is a number assigned sequentially to the display block.
[0079]
The display block is usually composed of a plurality of discussions. As described above, in the presentation, explanation by the presenter and discussion including other conference participants occur alternately. Therefore, the explanation and the discussion related thereto are defined as one discussion block, and one display block is divided and held for each discussion block.
[0080]
The “discussion block number” in the field 133 is a sequential number of the discussion block in the display block.
[0081]
The comment structure section that defines one discussion block, that is, the explanation and the discussion section related thereto is specified by the start comment in the field 134 and the end comment in the field 135. The message numbers shown in FIG. 6 are used as the message numbers of the start message and the end message in the fields 134 and 135. That is, the message numbers in the fields 134 and 135 are pointers to the message structure file.
[0082]
Next, the configuration of the local PC 40 will be described. FIG. 9 shows a configuration example of the local PC 40.
[0083]
The same contents as the large screen 1 are displayed on the display device 41 of the local PC 40. The conference participant can attach a memo to the screen content using the keyboard 7 or the pen input device 8 while referring to the screen. Such user input operation is detected by the user input information extraction unit 42, and the input time and position of the user input information, the input information, etc. are input log as an input log file in the form of a table as will be described later. It is recorded in the file storage unit 43. In this example, as for the pen input, user input information is recorded in units of strokes.
[0084]
The input object extraction device 44 extracts an input object as a series of input units from the user input information stored in the input log file storage unit 43, and inputs the input object file as an input object file in a table format as will be described later. Record in the storage unit 45.
[0085]
In the personal information management file generation unit 46, the information in the input object file storage unit 45 is converted into a hierarchical structure of partial structures in units of pages, display blocks, and discussion blocks in a form corresponding to the above-described conference information management file. And is recorded and managed in the personal information management file storage unit 47.
[0086]
FIG. 10 is an input example of the user's personal memo on the local PC 40. At the same time, this coincides with a display example when personal information is displayed as an overlay in presentation information reference processing at the time of meeting information playback described later.
[0087]
As described above, the presentation screen as shown in FIG. 10 is displayed on the display screen of the local PC 40, and the user can input a personal memo on this screen.
[0088]
In FIG. 10, 211, 212, and 214 indicate that the user has been marked by pen input. On the other hand, reference numeral 213 indicates that the user's personal memo is input as text information from the keyboard. A handwritten memo (character string) can also be input by pen input. However, since it does not have a character recognition function, the handwritten memo of the character string is processed as a set of strokes.
[0089]
FIG. 11 is a diagram for explaining the data structure of the input log table 140 as an input log file. As this input log file, the input from the keyboard 7 or the pen input device 8 is monitored by the user, and if the input is from the keyboard 7, the pen input device 8 may be a character string unit until the carriage return is input. For example, input is extracted and recorded in units of strokes.
[0090]
A field 141 of the input log table 140 is an input number, which is a sequential number for user input information. The next field 142 is a column for recording “type”, which is input from the keyboard 7 or input from the pen input device 8, and has a type of “text” and “stroke”, respectively.
[0091]
The “input item” in the next field 143 is a part for recording the substance of the input information. That is, in the case of text, the input character string is recorded here. In the case of a stroke, a series of coordinate lists representing the stroke are recorded.
[0092]
“Position coordinates” in the next field 144 represents the position where the input object is input. In the text, an insertion position indicated by a pointer or the like is extracted before input. In the stroke information, this position coordinate information is not extracted because it is included in the input item.
[0093]
In the next field 145 “start time”, the input start time of the corresponding user input information is recorded, and in the field 146 “end time”, the time when the input of user input information is completed is recorded. .
[0094]
FIG. 12 is a diagram for explaining the data structure of the input object table 150 of the input object file storage unit 45. In this example, the input object is extracted from the data of the input log file stored in the input log file storage unit 43, and the memo writing information for each individual is held for each unit object.
[0095]
Here, the input object refers to a group of strokes and text input within a certain period of time as a single unit.
[0096]
“No” in the field 151 of the input object table 150 is an input object number, which is a sequential number for identifying the input object. In the next field 152 “input object”, the type of the input object is recorded. In this example, there are two types, text or stroke. When the type is text, the value of each field is the same as the corresponding field value of the input log table 140 described above.
[0097]
In the “configuration item” of the field 153, a list of input numbers in the field 141 of the input log table 140 that is an element constituting the input object is recorded.
[0098]
In the “input start time” of the next field 154, when the input object is a stroke, the start time of the input data of the minimum input number among the strokes constituting the input object is recorded. In the case of text, the input start time is recorded. Similarly, in the “input end time” of the field 155, in the case of a stroke, the end time of the input data of the maximum input number among the strokes constituting the input object is recorded. In the case of text, the input end time is recorded.
[0099]
In the “area” of the next field 155, the coordinates of the display rectangular area when a series of stroke groups are grouped are recorded. In this example, a set of (starting point coordinates, diagonal coordinates) is recorded as the coordinate values of the display rectangular area.
[0100]
FIG. 13 is a diagram for explaining the data structure of the personal information management file (personal information management table) 160 in the personal information management file storage unit 47. In this embodiment, in the personal information management file, the input object information from each user is subdivided according to the display information and the structure of the discussion in the same manner as the conference information management table 130 as the conference information management file. Retained and managed in page units.
[0101]
The “page number”, “display block number”, and “discussion block number” in the fields 161 to 163 of the personal information management table 160 are the same as those in the conference information management table 130 as the conference information management file. In the “input object list” in the field 164, pointers to all input objects related to the time zone corresponding to the discussion block are held.
[0102]
Next, the conference information recording processing operation will be further described.
[0103]
FIG. 14 is a flowchart for explaining an outline of processing of the recording device unit of the conference information recording / reproducing device. In the process of the recording apparatus unit in this example, a file is generated by the five processes shown in FIG.
[0104]
FIG. 14A shows an outline of the flow of the conference audio data file recording process. The audio data input from the audio input device 5 or 9 such as a microphone is directly converted into an audio file by the conference audio data file recording process. It is stored in the conference audio data file storage unit 23 and recorded. In this example, this is the conference data itself.
[0105]
FIG. 14B shows an outline of the flow of the message structure extraction process. That is, in the speech structure extraction process, the speech structure table information stored in the speech structure file storage unit 27 is obtained from the speech data from the speech input devices 5 and 9 and the display page change notification information sent from the presentation tool 11. Generate.
[0106]
FIG. 14C shows an outline of the flow of display information extraction processing. In the display information extraction process, information related to time-series changes in the presentation display is extracted from the display page change notification information, and a display information table stored in the display information file storage unit is generated.
[0107]
FIG. 14D shows an outline of the flow of personal information extraction / management processing. That is, in the personal information extraction / management process, an appropriate input object is recognized from user input such as a personal note made on the presentation information by using the keyboard 7 or the pen input device 8 and the like, and in page units. A personal information management table for managing the personal information is generated and stored in the storage unit 47 as a personal information management file.
[0108]
FIG. 14E shows an outline of the flow of conference information extraction / management processing. In this conference information extraction / management process, information generated by other processes is used, and the message structure file in the message structure file storage unit 27 and the information in the display information file in the display information file storage unit 30 are displayed on the page. A process for generating a conference information management file for managing the conference information in a page unit and as a hierarchical structure is performed by restructuring into a unit management form. The conference information management table is stored in the conference information management file storage unit 33 as the conference information management file.
[0109]
FIG. 15 is a flowchart for explaining the outline of the message structure extraction process. First, in step 300, the page number currently displayed is set as the page number of the conference material as an initial setting.
[0110]
As described above, in this example, audio data having a sound volume of a certain level or higher, and audio data having the sound volume continued for a certain length of time or longer are extracted as speech. In step 301, it is determined whether the above-mentioned condition as a statement is satisfied. In the next step 302, the message number, the participant name, and the message start time, which are data of items corresponding to the fields of the message structure table 110 (FIG. 6), are extracted.
[0111]
In this case, a new sequential number is assigned as the speech number. The participant name is specified by searching the conference participant table 120 (FIG. 7) based on the input device number of the voice input device in which the speech is detected. In this example, the speech start time is a time that is a predetermined time before a point in time when it is detected that the audio data has exceeded a certain level. This is to prevent the beginning part of the statement from being lost.
[0112]
In the case of this example, there are two conditions for determining the end of speech, and each of the conditions is determined in step 303 and step 306.
[0113]
First, in step 303, if audio data of a certain level or lower continues for a certain period of time, it is regarded as the end of speech, and the speech end time is calculated in step 304. In step 305, the extracted information is recorded in the message structure table 110. After step 305, the process returns to step 301, and the speech section detection process as described above is repeated.
[0114]
Further, in this example, even when the utterance is continuous, it is determined that the utterance content has changed at the timing when the presentation page has changed, and it is determined that the utterance has ended. In step 306, it is determined whether or not there is a display page change notification. If there is a display page change notification, the process proceeds to step 307, where the speech end time is calculated.
[0115]
In this example, in step 307, the speech end time is a time obtained by adding a predetermined time to the page change time. As described above, the reason for adding the fixed time is to include the adjustment because there is a possibility that the speech is not clearly separated at the break of the page.
[0116]
In the next step 308, the extracted information is recorded in the message structure table 110. Then, the process proceeds to step 309 to perform processing for the next message data. That is, a new message number is assigned to the next message, and a time obtained by subtracting a predetermined time from the arrival time of the display page change notification is assigned as the message start time. Further, the page number after the change notification is assigned as the page number.
[0117]
The reason why the fixed time is subtracted from the arrival time of the display page change notification as the speech start time is the same as that for adjusting the speech end time. That is, there is a possibility that a discussion regarding the reason for the transition is included before the page is transitioned. In this case, it is necessary to set a longer adjustment time than the adjustment of the speech end time.
[0118]
FIG. 16 is a flowchart for explaining an overview of the display information extraction process. First, in step 351, a display information change notification indicating the start of presentation is received. In the next step 352, page number information is obtained from the display information change notification, and the timer time at that time is set as the display start time. Further, 1 is set as the initial value of the display number.
[0119]
In the next step 353, it is determined that a presentation end signal has been received from the presentation tool 11. If there is no notification of completion, the process proceeds to step 354, where it is determined whether or not a display page change notification has been received.
[0120]
When the display page change notification is received, the process proceeds to step 355, and the current timer time is set as the display end time as processing of information related to the display before the display page change. Then, in the next step 356, the information of the extracted display end time is recorded in the display information table 100.
[0121]
In the next step 357, processing for the next display data is performed. That is, a new page number included in the display page change notification is set for the page number, and the display end time of the previous data is substituted for the display start time. New display numbers are also assigned sequentially.
[0122]
If a presentation end notification is received in step 353, the process proceeds to step 358, and the display end time is set, information is recorded in the corresponding field of the display information table, and the process is ended, as in steps 355 and 356. To do.
[0123]
FIG. 17 is a flowchart for explaining the outline of the conference information extraction / management process.
[0124]
First, in step 401, display data corresponding to one display block is read from information in the display information table 100 of the display information file storage unit 30. In the next step 402, it is determined whether there is display data to be read from the display information file storage unit 30. If there is display data to be read, the process proceeds to step 403, and a new display block number is assigned to the read display data.
[0125]
After step 403, the process proceeds to step 404, where message structure data corresponding to the display data is extracted. The message structure data is read from the file, and while the page number is equal to the page number of the display data, the message structure data is repeatedly read to extract a series of continuous message structure data having the same page number.
[0126]
In the next step 405, speech structure division processing is performed on the extracted series of speech structure data, and the series of speech structure data is divided into a plurality of discussion blocks. Further, in the next step 406, the extracted discussion block group is recorded in the corresponding page number and display block number fields (records) of the conference information management table. The series of processing from step 401 to step 406 is repeated for all data in the display information file.
[0127]
FIG. 18 is a flowchart for explaining the outline of the message structure dividing process in step 405 of FIG. In this message structure dividing process, a process for dividing the extracted series of message structure data into several discussion blocks is performed. In this example, the discussion block is a series of speech structure sections consisting of an explanation of a presentation and an argument about questions and the like.
[0128]
In step 411, a new discussion block number is set as an initial value setting process. In the next step 412, one piece of utterance data is extracted from the utterance group of a series of utterance structure data. Then, in step 413, it is determined whether or not the processing has been completed for the data group of the message structure. If the processing has not been completed, the processing proceeds to step 414.
[0129]
In step 414, it is determined whether the statement is an explanation part or a discussion part of the presentation. In this example, when the speaker is a presenter and a speech having a speech interval time longer than a certain length, this speech is determined to be an explanatory speech. If this condition is not met, it is determined to be a discussion part.
[0130]
If it is determined that the utterance is not an explanatory utterance, the process returns to step 412 to proceed to processing of the next utterance in the utterance list.
[0131]
If it is determined in step 414 that the utterance is an explanatory utterance, the process proceeds to step 415. In this step 415, a series of utterances from the previous explanation utterance of the utterance determined to be the explanation utterance to the utterance before the explanation utterance being processed is extracted as one discussion block.
[0132]
In the next step 416, a new discussion block number is set for the processing of the next discussion block, and the explanation message data being currently processed is added to the statement list of the discussion block. Then, the process returns to step 412 and proceeds to the process of extracting the next discussion block.
[0133]
FIG. 19 is a flowchart for explaining the outline of the user input information extraction processing. This is a process for recording a history of input data in which the user has made notes and checks on the presentation material using the PC 40 at hand.
[0134]
First, in step 451, the user waits for input and checks whether or not there is an input. When an input from the user is generated, the time at which the input is generated is acquired from the timer 25 in step 452, and this is set as the input start time. In the next step 453, the instruction type of the user input is determined, and the process is branched according to each instruction type.
[0135]
If the instruction type is “input”, the process proceeds to step 454 to determine whether the user input is input from the keyboard or pen input device. If the input is a pen input, the process proceeds to step 455, where a stroke trajectory is received every unit time Δt, and a series of coordinate groups is substituted as stroke data into the input item field 143 of the input log table 140.
[0136]
In the case of keyboard input, the process proceeds from step 454 to step 456, and the coordinates of the point at which the input position is designated are extracted. In the next step 457, a series of input character strings from the keyboard is substituted as input data into the input item field 143 of the input log table. In this example, it is determined as a series of keyboard inputs from the start of input until a carriage return is input.
[0137]
Then, the process proceeds to step 458, and the time when the input is completed is obtained from the timer 25. In step 459, a new sequential number is assigned to the input number and recorded in the input log file storage unit 43.
[0138]
If it is determined in step 453 that the instruction type is “delete”, the process proceeds to step 460 to delete the instructed input data from the input log file. If it is determined in step 453 that the instruction type is “move”, the process proceeds to step 461 to change the position coordinates of the instructed input data. Further, when it is determined in step 453 that the instruction type is “copy”, new input data holding the same information as the designated input data is generated and recorded in the input log file 140.
[0139]
FIG. 20 is a flowchart for explaining the outline of the input object extraction process. As described above, in this example, the input object is extracted from the input log data.
[0140]
First, in step 471, input log data is read from the input log file. In the next step 472, it is determined whether all processing of the input log data has been completed. If all the processes are finished, the input object extraction process is finished.
[0141]
If the input log data has not been processed, the process proceeds to step 473, where it is determined whether the type of input data read is stroke input by a pen input device or text input by a keyboard. Different processing is performed depending on the type.
[0142]
Steps 474 to 479 are processes for extracting a series of stroke objects. A series of strokes refers to a group of data that is continuous stroke type input data and is input at intervals within a certain period of time.
[0143]
In step 474, the value of the input object type and the input start time are set. Here, the input start time is the input time of the first stroke of a series of strokes. Next, in step 475, the next input data is read.
[0144]
In the next step 476, it is determined whether or not the next input data exists, and if not, the process proceeds to step 491 in FIG. 21 for the end processing procedure when the data is ended.
[0145]
In this step 491, a value is set with the end time of the input data being processed as the input end time of the input object. In the next step 492, the coordinates of the rectangular area including all of the series of strokes included in the extracted input object are calculated as the display rectangular area. In the next step 493, a new input object number is assigned to the extracted stroke object, the input object file (input object table) 150 is recorded in the storage unit 45, and the process is terminated.
[0146]
If it is determined in step 476 that the next input data exists, the process proceeds to step 477. In this step 477, it is determined whether or not the data type of the read input data is a stroke. This is to determine whether the stroke data continues in order to extract a series of strokes.
[0147]
If the data type determined in step 477 is other than stroke data, it is determined that a series of stroke objects has been extracted, and the process proceeds to step 494 in FIG. I do.
[0148]
That is, the processing from step 494 to step 496 is a post-processing procedure when a normal series of stroke objects is extracted. The processing is the same as the processing from step 491 to step 493. After the processing is completed, the processing proceeds to step 473 in order to continue processing the input data being processed.
[0149]
If the data type determined in step 477 is a stroke, the process proceeds to step 478, where it is determined whether the input data is continuous input data depending on whether the input time interval is within a certain time. Is judged. That is, it is determined whether the time difference between the end time of the previous input data and the start time of the input data being processed falls within a certain unit time Δt.
[0150]
If there is an interval of a certain time or more in the time difference, it is determined that a stroke object has been extracted, and the process proceeds to step 494. If there is no interval of a certain time or more in the time difference and the stroke object continues, in the next step 479, the input data being processed is added to the configuration item, and the next input data is processed. Return to step 475 to perform.
[0151]
Next, processing when the type of input data is text will be described. When the input type is text, the input data matches the input object. Therefore, in step 480, the value of the corresponding item of the input data is substituted for each item of the input object.
[0152]
In the next step 481, a text display rectangular area is calculated. In step 482, a new object number is assigned to the extracted text object and recorded in the input object file. After step 482, the process returns to step 471.
[0153]
When the input object extraction process is completed as described above, a personal information management file is generated from this information. This personal information management file has the same format as the conference information management file. An input object group whose input start time is within the time interval of the discussion block is held in the input object list field of the corresponding discussion block.
[0154]
[Description of the playback device portion of the conference recording / playback device]
FIG. 1 is a block diagram showing a configuration of a playback device portion for accessing conference information desired by a user after the conference ends in the conference information recording / playback device according to the embodiment of the present invention.
[0155]
In this embodiment, the conference information is reproduced from a specific page of the presentation tool by accessing a chart that visualizes the corresponding statement structure. It features the method. There are a number of ways to access this particular message structure.
[0156]
First, one of them is a case where a message structure diagram corresponding to an arbitrary page of the conference material is displayed with reference to the screen on which the conference material is presented. In the case of this example, the conference material is a presentation material, and all the comments related to the presentation material page of the presentation currently being referenced can be referred to and reproduced.
[0157]
Furthermore, by using together with personal information such as memos recorded as user input information by the conference participants, only the utterance structure section near the memo writing is targeted, and only the utterance structure diagram of that part is displayed. Only the meeting information corresponding to it can be reproduced. Rather than identifying the page you want to refer to from presentation materials or notes, you can grasp the discussion status for each page, and from that discussion status, identify any referenced page and access the relevant meeting information You can also
[0158]
In FIG. 1, in this embodiment, the display device 51 displays the contents of the presentation material used during the presentation in the meeting. In addition, as described above, the conference participants may also display the entered memo contents in the conference. On the screen of the display device 51 at this time, the same display screen as in the presentation shown in FIG. 10 is displayed.
[0159]
The presentation control display unit 53 is a device for controlling the display of the presentation content in the display device 51. Basically, the contents of each page of presentation presentation materials as conference materials stored in the presentation file storage unit 58 are displayed.
[0160]
When there is user input information of individual conference participants, the presentation control display unit 53 superimposes the page contents of the presentation material in the presentation file storage unit 58 on the personal information display unit 56 and the memo information attached thereto. Control is performed so that the combined items are displayed on the display device 51.
[0161]
The instruction input device 52 inputs an instruction for search on the display screen of the display device 51. That is, in this playback apparatus unit, as will be described later, when a searcher instructs an area corresponding to a message to be played back using a pointing device such as a mouse, the corresponding audio data is played back.
[0162]
When a request for access to the statement structure diagram is instructed by the user through the instruction input device 52 while referring to the presentation content, the request detection unit 59 detects the request. The display status determination unit 60 determines the display status of the current presentation presentation material page and the like and the selection status of the handwritten memo.
[0163]
In this example, the display information detection unit 61 detects information regarding a display page, a selected object, and the like via the presentation display control unit 53. Then, the display information detection unit 61 sends the detected information to the display status determination unit 60. The display status determination unit 60 determines what speech section should be displayed. The display status determination unit 60 determines, for example, whether to collect all conference information sections related to a page or to extract an arbitrary partial structure.
[0164]
The speech structure section detection unit 62 receives the determination result of the display state determination unit 60 and automatically determines the speech structure section according to the corresponding display state or object selection state. The utterance structure section extraction unit 63 receives the information of the utterance structure section automatically detected by the utterance structure section detection unit 62 and performs processing for extracting the corresponding utterance structure section from the utterance structure file storage unit 27.
[0165]
The utterance structure diagram generation unit 64 generates a utterance structure diagram that is a chart diagram of the utterance structure corresponding to the utterance structure interval extracted by the utterance structure interval extraction unit 63. The generated message structure diagram is displayed on the display device 51 by the message structure diagram display control unit 54.
[0166]
Instead of specifying the conference information to be referred to from the presentation content, for example, it is necessary to provide a means for specifying the page to be referred from the discussion state, such as wanting to reproduce the discussion content of the place where the discussion was exciting The discussion status display control unit 55, the discussion status determination unit 65, the discussion information extraction unit 66, and the discussion activation level determination unit 67 are processing units related to the display of the discussion status.
[0167]
The discussion state determination unit 65 discriminates between the explanation section and the discussion section from the information in the comment structure file storage section 27, and extracts the discussion section. The method of detecting the discussion section is the same as described above. A speech section having a predetermined length or more is detected as an explanation speech, and a discussion section from one explanation speech to the comment before the next explanation speech is used as the discussion section. Determine and extract.
[0168]
Regarding the extracted discussion section, the discussion information extraction unit 66 extracts attributes related to the discussion, such as the number of utterances, the number of speakers, and the discussion time. The discussion activation level determination unit 67 calculates the activity level of the discussion from these extracted attribute values, and determines the activation level. The discussion status display control unit 55 visualizes and displays discussion attribute information and activation level for each page.
[0169]
FIG. 23 is a diagram illustrating an example of a speaker chart that visualizes a speech structure as an example of a speech structure diagram.
[0170]
Reference numeral 201 denotes a speaker chart display area. The speaker chart is composed of two areas: a total meeting time display area 202 that shows an overview of the entire meeting, and a message structure display area 203 that displays details of the part of the message structure corresponding to the detailed display location 204.
[0171]
The message structure display area 203 includes an area 206 for displaying a speaker name for identifying a speaker, and a message structure diagram display area 207 for visually displaying a state of transition of the speaker. Each column of the chart indicates when and how much time each conference participant made a speech during the conference time. In addition, by looking at the speech structure for all the conference participants, it is possible to read the speech transition structure indicating who has transitioned from what speech.
[0172]
In addition, by designating an arbitrary position from the speaker chart, the corresponding audio data can be reproduced. A triangular point 205a in the all meeting time display area 202 or a broken line 205b in the message structure display area 203 indicates the position on the speaker chart corresponding to the audio data being reproduced at that time.
[0173]
FIG. 24 is a flowchart for explaining the outline of the reproduction process when referring to the conference information.
[0174]
First, in step 501, a reference type of conference information is determined. The reference type is divided into a presentation type and a discussion situation type. In the case of the presentation type, the process proceeds to step 502, and the user specifies a page to be accessed as appropriate according to the content while referring to the content of the presentation.
[0175]
In the next step 503, it is determined whether or not there is an access request for displaying the message structure diagram from the user. When a request to display the message structure diagram is received, in step 504, a message structure section according to the presentation display status at that time is specified, and a chart of the message structure diagram is generated. In the next step 505, a chart of the generated message structure is displayed, and an arbitrary message instructed by the user is reproduced from the chart.
[0176]
If the reference type determined in step 501 is the discussion status type, the process proceeds to step 506, and information summarizing the discussion status is visualized and displayed. The user specifies a page to be referred to from the screen displaying the discussion status. When the user specifies a page to be referred to, the user's instruction is referred to in step 507, and if the user instruction requests access to the speech structure after referring to the presentation content, the process proceeds to step 502. move on. If the user instruction directly requests access to the message structure diagram, the process proceeds to step 504.
[0177]
FIG. 25 is a flowchart for explaining the outline of the presentation information reference process in step 502 of FIG. Here, referring to the presented material, accessing the chart structure of the speech structure from there is a series of procedures for reproducing the conference information.
[0178]
First, in step 511, a presentation display request is waited. If there is a request, the process proceeds to step 512, and the presentation tool 11 displays the contents of the presentation data. In the next step 513, the displayed page number is acquired from the presentation tool 11. In the next step 514, the input object group related to the acquired page number is read from the personal information management file storage unit 47.
[0179]
In the next step 515, the input object group of personal information is displayed on the already displayed presentation screen. This provides a situation where the material presented at the conference and the personal memo can be referenced simultaneously.
[0180]
In the next step 516, an instruction input from the user referring to the material is awaited, and corresponding processing is performed depending on the type of instruction input.
[0181]
First, when the instruction input from the user is an instruction to end presentation reference, the process is ended as it is. If the instruction input from the user is an instruction to refer to another page, the process returns to step 512, and the presentation data information and the personal information are superimposed and displayed in the same manner as described above. Further, when the instruction input from the user is a message structure display request, the process proceeds to step 517 to execute a message structure access process.
[0182]
FIG. 26 is a flowchart for explaining the outline of the message structure access processing in step 517. This speech structure access process is to extract an appropriate speech structure section according to the display state and selection state of the presentation.
[0183]
In step 521, first, it is selected whether to automatically determine the display / selection status, or whether to give a hierarchical level instruction explicitly requesting display by the user.
[0184]
In the case of automatic selection, the process proceeds to step 522, where the selection state of whether or not the input object is selected is determined, and the type for selecting the speech section is selected in steps 523 and 524 according to the determination result. The value of the variable to represent is set.
[0185]
In other words, if no object is selected, the process proceeds to step 523, where all the speech segment sets related to the entire displayed page are extracted. Therefore, the page is set as the type.
[0186]
On the other hand, if the input object is selected, the process proceeds to step 524, and in this case, the speech section related to the discussion block is extracted. Therefore, a discussion block is set as the type. In this case, a speech structure section of an appropriate discussion block corresponding to the time when the input object is input is selected.
[0187]
In the next step 525, a page number and an object number necessary for extracting an appropriate speech structure section are set. The page number has already been acquired in the presentation information reference information processing of FIG. The object number is set when an object is selected. If no object is selected, nil is set.
[0188]
In the next step 526, an appropriate speech structure section is extracted according to the set status variable, a chart diagram is generated, and visualization information is generated. Then, in the next step 527, the chart is displayed on the display screen of the display device 51. In this chart, the user can issue a playback instruction for an arbitrary statement.
[0189]
If it is determined in step 521 that the instruction is a hierarchical level instruction explicitly requested to be displayed by the user, in step 531, a hierarchical level instruction input from the user is determined. Then, according to the value of the instruction input from the user, the values of the page, the display block, and the discussion block are set as the display and selection status types in steps 5332, 533, and 534, respectively, and the process proceeds to step 525. The processing after this step 525 is the same as in the automatic selection described above.
[0190]
FIG. 27 is a flowchart for explaining the outline of the speech structure visualization process.
[0191]
First, in step 601, the type of display and selection situation is determined. Then, the corresponding visualization process is performed according to the determined type. That is, when the display / selection status type is page, the process proceeds to step 602 and subsequent steps, and all the speech structure sections related to the presentation display are extracted and simultaneously displayed as a speech structure diagram.
[0192]
First, in step 602, a conference information management file relating to the corresponding page is read. This conference information management file holds all the information related to the message structure related to this page. In the next step 603, the speech time is calculated in order to classify the size of the display area according to the length of the speech time. First, the total speech time of all the extracted speech structure sections is calculated. At the same time, the speech time for each display block is also calculated.
[0193]
In the next step 604, with respect to the area of the message structure display area 203 shown in FIG. 23, the display area area is set according to the ratio between the total time for each display block and the total page display time. The chart is assigned to each display area. In step 605, in order to indicate in which time zone each display block exists in the entire meeting, the entire meeting time display area 202 shown in FIG. 23 corresponds to each display block. Displays the time zone.
[0194]
FIG. 30 is a diagram for explaining a display example of the utterance structure diagram in page units in this example. All remark structure diagrams related to the selected page are displayed in units of display blocks. As a result, a list of discussions related to the page can be displayed.
[0195]
In FIG. 30, reference numeral 701 denotes an all-conference time display area, and reference numerals 701a, 701b, and 701c denote time periods when the selected page is displayed. Reference numeral 703 denotes a message structure diagram display area. Reference numeral 702 denotes display block areas 702a, 702b, and 702c corresponding to the respective time zones 701a, 701b, and 701c in the message structure diagram display area.
[0196]
703a is a remark structure diagram related to the display block displayed in the time zone 701a. 703b is a remark structure diagram related to the display block displayed in the time zone 701b. 703c is a remark structure diagram related to the display block displayed in the time zone 701c.
[0197]
Next, when it is determined in step 601 of FIG. 27 that the type is a discussion block, processing from step 611 to step 614 in FIG. 28 when the type of the display / selection state is a discussion block is executed.
[0198]
First, in step 611, a discussion block including an input object number is extracted from the personal information management file. In the next step 612, a speech section corresponding to the extracted discussion block is extracted. The extracted speech section is visualized and displayed so as to be within the speech structure diagram display area. Next, in step 613, as in step 605, the time zone corresponding to the discussion section is shown in the all conference time display area. In the next step 614, marking indicating the position of the selected input object is performed on the chart structure of the displayed message structure.
[0199]
FIG. 31 is a diagram for explaining a display example of the utterance structure diagram in block units. Reference numeral 711 denotes a total meeting time display area, and 711a shown shaded indicates a time zone corresponding to the discussion section.
[0200]
In the message structure diagram display area 712, the most different information from the above-mentioned display in page units is information on the selected input object. That is, in the figure, 712a shown shaded indicates a time zone in which the input object was input. A triangular mark 712b indicates the input start time of the input object.
[0201]
If it is determined in step 601 that the display / selection type is display block, the processing from step 621 to step 624 shown in FIG. 29 is executed. This process is the process from step 611 to step 614 in FIG. 28, in which the discussion block is replaced with the display block.
[0202]
FIG. 32 is a diagram for explaining an example of presentation of the discussion status displayed on the screen of the display device 51 by the discussion status display control unit 55 and access to the comment structure diagram from the discussion status.
[0203]
In FIG. 32, reference numeral 801 denotes an example of an active status display screen. Reference numeral 802 denotes an example of a display screen for the presentation contents and personal information of the designated page when the page is designated. Reference numeral 803 denotes a display screen for the message structure diagram described above.
[0204]
In the discussion status activation status display screen 801 in FIG. 32, according to the discussion activation level, the rank of each page is displayed according to the shade density of the area 801b of each page. Yes. For example, it can be seen that pages 1, 7, and 9 were actively discussed. Therefore, a page to be referred to can be selected from the display screen 801.
[0205]
In this example, the activity of discussion is defined and calculated as a discussion function, the number of speakers, and a composite function of the number of speakers. In this example, as shown in the explanation column 804 of FIG. 32, the activity value of this argument is divided into five levels, and the activation level is determined. Reference numeral 801a on the display screen 801 is a reference index of these five activation levels.
[0206]
When the page of the conference material is specified, the content of the page is displayed as a display screen 802 as in a path 805 indicated by an arrow, and the subsequent processing is the same as the access method described so far.
[0207]
On the other hand, as described above, it is also possible to directly display the message structure diagram without referring to the contents of the conference material, which is indicated by a path 806. This is because if the person who created the conference material searches the conference information after the conference ends, the content of the conference material may be familiar, and there is no need to display the content of the conference material. It is a thing.
[0208]
In this example, the activation level is exemplified for the discussion activation status, but the same effect can be obtained by directly graphing attribute values such as the discussion time and the number of speakers.
[0209]
In the above embodiment, since the conference material is managed in units of pages, the page number is used as the material identification information, but there are a plurality of conference materials as in the case of a conference on a plurality of themes. If it can be classified into types, material identification information may be assigned to each conference material for each theme across multiple pages. In this case, it is of course possible to provide the management information for each page by giving identification information for each page to the conference material of a plurality of pages for each theme.
[0210]
Therefore, the material identification information is not limited to the page number, and various identification information can be used.
[0211]
Moreover, although the above embodiment is an example when this invention is applied to a presentation type meeting, it cannot be overemphasized that this invention is not limited to this type of meeting.
[0212]
In addition, the conference materials are not limited to those displayed on the display screen. Even if the conference materials are presented as OHP sheets or distributed at the conference, the page numbers, conference material numbers, etc. If the identification information is given and managed, the present invention is applicable.
[0213]
In the above description, the discussion block is detected with the explanation speech as the partial speech structure. However, active discussion is performed in the speech section where a plurality of conference participants frequently change speakers. Various other methods can be used as a method for detecting a partial utterance structure, such as detection as a discussion utterance section.
[0214]
In the above-described embodiment, the message structure file, the conference information management file, and the personal information management file are generated at the time of recording and stored in the respective storage units. It is also possible to reproduce the entire information and generate it at that time.
[0215]
Further, in the above-described embodiment, the case where only audio information is recorded as the conference information has been described. However, the present invention can also be used for access when recording video information such as a conference scene at the same time and reproducing it. Is applicable.
[0216]
【The invention's effect】
As described above, according to the speech structure information presentation device according to the first aspect of the present invention, the structure of all discussions related to a specific conference material is visualized as a speech structure diagram. Compared with the data displayed in order, it is easier to understand the entire discussion of the document content. Therefore, when accessing the content of the conference, it is easy to access the necessary remarks because it is possible to narrow down which remarks should be accessed by grasping the entire context.
[0217]
In addition, according to the speech structure information presentation device according to the second aspect of the present invention, a partial structure can be extracted from conference information having no structure, data can be handled in units of partial structures, and a partial structure exists. The search processing efficiency is improved as compared with the case where the search is not performed.
[0218]
Further, according to the speech structure information presentation device of the invention of claim 3, when a specific time zone is designated by the searcher of the conference information, the speech structure section at each level can be easily extracted. Thus, the generation efficiency of the utterance structure diagram is improved.
[0219]
Further, according to the speech structure information presentation device of the invention of claim 4, it becomes possible to easily associate the conference material with the partial structure of the speech structure, and the search by handwritten memo and the search by the speech structure Can be effectively combined, so that the processing efficiency for accessing the message structure is improved.
[0220]
According to the speech structure information presentation apparatus of the invention of claim 5, when the search target is ambiguous, the level and time zone of the identification information unit of the conference material are narrowed down to some extent. When the level of each display section and the location of the utterance are narrowed down, the partial structure level such as the discussion utterance section can be designated, so that the search access can be performed as intended by the user. In this way, the search level can be selected according to the level of clarity of the target to be searched, and efficient narrowing down is possible.
[0221]
According to the utterance structure information presentation apparatus of the invention of claim 6, the utterance section at an appropriate utterance structure level is obtained as a search instruction input even if there is no explicit instruction from the user. Can only present. As a result, the user's search instruction operation is simplified, unnecessary information does not have to be presented, and the load on the user to access the necessary information is reduced.
[0222]
Furthermore, according to the speech structure information presentation device of the invention of claim 7, when specifying the reference conference material, not only from the content of the material, but also from the discussion status for the material, the active part of the discussion is accessed, etc. You can easily access important discussion parts without referring to all meeting information. Moreover, this information can shorten the time for grasping | ascertaining the content of a meeting for the person who did not participate in a meeting.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a playback device section of a conference information recording / playback device to which an embodiment of a speech structure information presentation device according to the present invention is applied.
FIG. 2 is a conceptual diagram showing a state of a meeting to which the present invention is applied.
FIG. 3 is a block diagram showing a configuration example of a conference recording PC of the recording device unit of the conference information recording / reproducing device to which the embodiment of the speech structure information presentation device according to the present invention is applied.
FIG. 4 is a block diagram showing a configuration example of a presentation PC of the recording device unit of the conference information recording / reproducing device to which the embodiment of the speech structure information presentation device according to the present invention is applied.
FIG. 5 is a diagram for explaining a data structure of a display information file used in the embodiment of the message structure information presentation device according to the present invention;
FIG. 6 is a diagram for explaining the data structure of a statement structure table used in the embodiment of the statement structure information presentation device according to the present invention;
FIG. 7 is a diagram for explaining a data structure of a conference participant table used in the embodiment of the speech structure information presentation device according to the present invention.
FIG. 8 is a diagram for explaining the data structure of a conference information management file used in the embodiment of the speech structure information presentation device according to the present invention;
FIG. 9 is a block diagram showing a configuration example of a local PC of the recording device unit of the conference information recording / reproducing device to which the embodiment of the speech structure information presentation device according to the present invention is applied.
FIG. 10 is a diagram showing a display example on the display screen of the conference information recording / reproducing device to which the embodiment of the message structure information presentation device according to the present invention is applied.
FIG. 11 is a diagram for explaining the data structure of an input log file used in the embodiment of the message structure information presentation device according to the present invention;
FIG. 12 is a diagram for explaining the data structure of an input object file used in the embodiment of the speech structure information presentation device according to the present invention;
FIG. 13 is a diagram for explaining the data structure of a personal information management file used in the embodiment of the speech structure information presentation device according to the present invention;
FIG. 14 is a flowchart for explaining the outline of the process of the conference information recording unit at the time of the conference in the conference information recording / reproducing device to which the embodiment of the message structure information presentation device according to the present invention is applied;
FIG. 15 is a flow chart for explaining the outline of the message structure extraction process in the conference information recording / reproducing apparatus to which the embodiment of the message structure information presentation apparatus according to the present invention is applied;
FIG. 16 is a flowchart for explaining an overview of display information extraction processing in the conference information recording / reproducing apparatus to which the embodiment of the message structure information presentation apparatus according to the present invention is applied;
FIG. 17 is a flowchart for explaining an outline of conference information extraction / management processing in the conference information recording / reproducing device to which the embodiment of the message structure information presentation device according to the present invention is applied;
FIG. 18 is a flowchart for explaining the outline of the speech structure dividing process in the conference information recording / reproducing apparatus to which the embodiment of the speech structure information presenting apparatus according to the present invention is applied;
FIG. 19 is a flowchart for explaining an overview of user input information extraction processing in the conference information recording / reproducing apparatus to which the embodiment of the message structure information presentation apparatus according to the present invention is applied;
FIG. 20 is a part of a flowchart for explaining an outline of input object extraction processing in the conference information recording / reproducing apparatus to which the embodiment of the message structure information presentation apparatus according to the present invention is applied;
FIG. 21 is a part of a flowchart for explaining an outline of input object extraction processing in the conference information recording / reproducing apparatus to which the embodiment of the speech structure information presentation apparatus according to the present invention is applied;
FIG. 22 is a part of a flowchart for explaining an outline of input object extraction processing in the conference information recording / reproducing apparatus to which the embodiment of the message structure information presentation apparatus according to the present invention is applied;
FIG. 23 is a diagram illustrating an example of a chart visualizing a message structure.
FIG. 24 is a flowchart for explaining the outline of reproduction processing when referring to conference information in the conference information recording / reproducing device to which the embodiment of the message structure information presentation device according to the present invention is applied;
FIG. 25 is a flowchart for explaining the outline of presentation information reference processing in the conference information recording / reproducing apparatus to which the embodiment of the speech structure information presentation apparatus according to the present invention is applied;
FIG. 26 is a flowchart for explaining the outline of the statement structure access process in the conference information recording / reproducing apparatus to which the embodiment of the statement structure information presentation device according to the present invention is applied;
FIG. 27 is a part of a flowchart for explaining the outline of the speech structure visualization process in the conference information recording / reproducing apparatus to which the embodiment of the speech structure information presentation apparatus according to the present invention is applied;
FIG. 28 is a part of a flowchart for explaining the outline of the speech structure visualization process in the conference information recording / reproducing apparatus to which the embodiment of the speech structure information presentation apparatus according to the present invention is applied;
FIG. 29 is a part of a flowchart for explaining the outline of the speech structure visualization process in the conference information recording / reproducing apparatus to which the embodiment of the speech structure information presentation apparatus according to the present invention is applied.
FIG. 30 is a diagram for explaining a display example of a utterance structure diagram for each page in the conference information recording / reproducing apparatus to which the embodiment of the utterance structure information presentation apparatus according to the present invention is applied.
FIG. 31 is a diagram for explaining a display example of the utterance structure diagram of the partial structure unit in the conference information recording / reproducing apparatus to which the embodiment of the utterance structure information presentation apparatus according to the present invention is applied;
FIG. 32 is a diagram for explaining a presentation example of the discussion status in the conference information recording / reproducing apparatus to which the embodiment of the statement structure information presentation device according to the present invention is applied, and access to the statement structure diagram from the discussion status; is there.
[Explanation of symbols]
3, 7 keyboard
4, 8 Pen input device
5, 9 Microphone as a voice input device
10 PC for presentation
20 PC for meeting recording
23 Conference audio data file storage
24 Utterance data extraction unit
25 timer
26 Statement structure table generator
27 Statement structure file storage
28 Notification detector
29 Display information extractor
30 Display information file storage
31 Partial structure detector
32 Management file generator
33 Meeting information management file storage
40 hand PC
42 User input information extraction unit
43 Input log file storage
44 Input object extraction unit
45 Input object file storage
46 Personal information management file generator
47 Personal information management file storage
51 Display device
52 Instruction input device
53 Presentation Display Control Unit
54 Remark Structure Diagram Display Control Unit
55 Discussion Status Display Control Unit
56 Personal information display
58 Presentation file storage
59 Request detector
60 Display status determination unit
61 Display information detector
62 Remark structure section detector
63 Remark structure section extractor
64 Statement structure diagram generator
65 Discussion State Judgment Unit
66 Discussion Information Extraction Unit
67 Discussion Activation Level Judgment Unit

Claims (8)

First storage means for storing material identification information for identifying each of a plurality of conference materials to be discussed in a conference, and presentation time interval information for each of the plurality of conference materials ;
Second storage means for storing time information of each speech section of a plurality of conference participants detected from conference audio information and the material identification information of the conference material presented when each speech was made When,
Material discriminating means for discriminating a conference material selected and instructed by the user among the plurality of conference materials;
For the determined meeting material in said article identifying means, based on said first memory means and information of said second storage means, in the meeting, and time period in which the conference material is presented, is the conference material and a remark structure for representing the occurrence status of statement interval of the conference participants in the presentation time interval, the speech structure diagram generating means for generating a visual Satoshika information,
An utterance structure information presentation device comprising: In the statement structure information presentation device according to claim 1,
From the mutual relationship between the information related to the presentation status of the conference material stored in the first storage means and the speech sections of the plurality of conference participants stored in the second storage means A utterance structure information presentation device comprising partial utterance structure detection means for extracting the partial structure of the utterance. In the statement structure information presentation device according to claim 2,
A first utterance structure level for extracting all utterance sections related to each of the conference materials, a second utterance structure level for extracting all utterance sections in a specific presentation time zone of the conference materials, and the partial utterance structure detection A utterance structure information presentation device comprising: a utterance structure management means for managing the utterance structure in a hierarchical structure including a third utterance structure level for extracting a section related to a specific topic detected by the means. In the statement structure information presentation device according to claim 3,
Third storage means for storing user input information, such as memos and markings, entered by each conference participant at the conference, for each conference participant;
Personal information management means for maintaining a relationship between the user input information and the first to third message structure levels managed by the message structure management means, and managing the user input information as a hierarchical structure;
An utterance structure information presentation device comprising: In the statement structure information presentation device according to claim 4,
A message structure that specifies and extracts a message structure section related to the specified user input information according to the specified hierarchy level by specifying the user input information and specifying a level of the hierarchy structure An utterance structure information presentation device comprising section extraction means. In the statement structure information presentation device according to claim 4,
A display unit that overlays and displays the content of the conference material presented in the conference and the user input information;
A remark of any one of the first to third remark structure levels according to a reference state of the conference material of the search instruction input by the user on the display means or a selection state of the user input information It is characterized by comprising speech structure section detecting means for automatically determining a section and detecting only a speech structure section connected to an arbitrary part according to the speech structure level in the presentation time section of the conference material. Remark structure information presentation device. In the statement structure information presentation device according to claim 1,
Explanation speech detection means for detecting, as explanation speech, a length of the speech section stored in the second storage means for a predetermined time or longer;
A discussion state determination means for extracting a comment structure of a discussion by deleting an explanatory comment from the comment structure;
Discussion information extraction means for extracting discussion attribute information such as the number of speakers and discussion time from information on the speech structure of the discussion;
Based on the discussion attribute information extracted by the discussion information extraction means, a discussion activation level determination means for calculating a discussion activity level indicating a rising situation of the discussion and determining a discussion activation level;
Means for visually displaying the climax of the discussion on the display screen according to the determination result of the discussion activation level determination means;
An utterance structure information presentation device comprising: A speech structure information presentation method realized by storage means, material discrimination means, and speech structure diagram generation means,
The storage means detects each presentation time interval of a plurality of conference materials to be discussed in a conference, and detects the detected presentation time interval information, and material identification information for identifying each of the plurality of conference materials Is stored in association,
Detecting the time of each speech section from the sound information of a plurality of conference participants of the conference, and time information of the detected speech interval, and the article identification information of conference material was being presented when each utterance is made and that Symbol憶process be stored in association with,
The material determining means determines a meeting material selected and instructed by the user from among the plurality of meeting materials,
The talk structure diagram generating means for conference material the discriminated by the article determination step, on the basis of the stored information in said storing step, in the meeting, the time and interval in which the conference material is presented, the conference and a remark structure for representing the occurrence status of statement interval of the conference participants in article presented time interval, the speech structure diagram generating step of generating a visual Satoshika information,
A speech structure information presentation method comprising:

Images