JPH11259501A - Speech structure detector/display - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a speech structure detection/display capable of displaying the relation of speeches having high interaction with each other. SOLUTION: A speech section detection means 11 detects the speech section of each speaker from voice signals from voice input means and a posture detection means 6 detects the posture of each speaker. A voice/posture recording means 13 records voice information from the voice input means records the information of the speech section of each speaker detected in the speech section detection means and the posture of each speaker detected in the posture detection means in correspondence. A speech object person specifying means 21 specifies a person to whom the speech is mode based on the information recorded in the voice/posture recording means. A speech flow detection means 22 detects the flow of the speech in a prescribed time section based on the recorded information of the vice/posture recording means and the result of the speech object person specifying means. Display information corresponding to the detected result of the speech flow detection means 22 is displayed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a statement structure detecting and displaying device for detecting and displaying the structure of a statement of a participant in a conference.

[0002]

2. Description of the Related Art If the structuring of a statement can be understood in a broad sense, there are techniques for structuring the statement by associating the statement with other information, and techniques for structuring the statement from the statement information itself. .

The former prior art is disclosed in Japanese Patent Laid-Open No. 6-34.
No. 3146, Japanese Unexamined Patent Publication No. Hei 7-226931, Japanese Unexamined Patent Publication No. Hei 6-205151, Japanese Unexamined Patent Publication No. Hei 6-176171, Japanese Unexamined Patent Publication No. Hei 7-182365, Marqee:
A Tool For Real-Time Video
o Logging (CHI'94 Human Fa
ctors in Computing System
s) and the like.

[0004] In these conference systems, while multimedia information such as audio / video of a conference is recorded,
A mechanism is provided to record input information such as pen input and keyboard input of meeting participants and the input time, and to be able to reproduce multimedia information related to the input information later using the input time. I have. This is a technique that does not structure the statement information but associates the statement information with the user input information.

[0005] Commercially available cassette tape recorders and mini-disc recorders also have a tag (track mark) that can be used to make it easier to retrieve important parts later when important information is recorded during recording. Some of them are provided with buttons, but this kind of technology is also regarded as a related prior art.

On the other hand, there is an approach of performing speech recognition as the latter, that is, a technique for structuring a speech from the speech information itself. It is emitted in a scene such as a conference or a lecture,
At present, it is extremely difficult to recognize and understand natural human speech. Therefore, for example, using a technique such as the word spotting method, the keyword is detected,
The result is structured using the result.

[0007] There is also a method of visualizing the state of the utterance, thereby making it easier for a human to grasp the state of the utterance. In this technology, a machine does not structure utterances, but is positioned as a technology that assists humans in structuring utterances.

A typical example is disclosed in Japanese Patent Application Laid-Open No. 8-317.
There is an electronic conference device described in JP-A-365-365. In this device, the magnitude of the recorded amount of each utterance of each speaker is taken on the horizontal axis,
By providing a graph display area indicating the order of conversation on the vertical axis, it is easy to visually grasp the state of speech. Since then
A diagram that has the concept of a time axis and further shows the state of speech is called a speaker chart.

A similar speaker chart is also found in research papers. For example, "CHI'95 MOS
AIC OF CREATIVITY ", Donald G. A research paper by Kimber et al., “Speaker Segmentation f
or Browsing Recorded Audi
"o" also describes the speaker chart information.

[0010]

In a meeting, it is convenient to be able to display the relations between the related statements on a speaker chart. For example, conference participant A
When an opinion or a question is asked to another conference participant B, if an interactive scene in which the conference participant B answers or refutes it can be determined from the chart, there is some discussion there. It turns out that there was
A searcher of the recorded meeting information is expected to be able to easily search for a discussion part to be reproduced using the clue as a clue.

Particularly, in a multi-person meeting with three or more people, there is a speech structure section having a high interaction as described above, and in other cases, a speech section in which speakers move in a quiet manner. In such a case, if the speech structure section having a high interaction as described above can be easily detected, it is expected that access to the important conference information portion becomes relatively easy.

However, the conventional conference system described above does not structure the relationship between a plurality of remarks. That is, in the former case of the conventional conference system, it is possible to structure the voice information in association with the user input information, but it is not possible to extract the flow of the utterance including the mutual relation between the utterances.

Further, even if an important word is picked up by a technique such as word spotting, it is extremely difficult to detect the flow of the above-mentioned utterance. It cannot be realized without the ability.

Further, in the conventional speaker chart, although the alternation of the speakers and the length of the speech can be known, the relevance and flow of each speech cannot be understood. For example, if one person speaks and then another person speaks, it can be grasped by the conventional speaker chart, but whether there was a flow of communication such as answering questions, or simply Whether the speaker has moved and started a new conversation flow cannot be known from the conventional speaker chart.

In view of the above, an object of the present invention is to provide an utterance structure detecting and displaying device capable of displaying a high mutual interaction between utterances.

[0016]

In order to solve the above-mentioned problems, a speech structure detecting and displaying apparatus according to the present invention comprises a speech input means for picking up a speech of a speaker and a speech signal from the speech input means. Utterance section detection means for detecting a utterance section of each utterer, attitude detection means for detecting an attitude of each utterer, and voice information from the voice input means, and detection by the utterance section detection means. The voice / posture recording means for recording the information of the utterance section for each speaker and the posture of each speaker detected by the posture detection means in association with each other, and the voice / posture recording means Based on the information, the utterance target person specifying means for specifying to whom the utterance is directed, the flow of the utterance in a predetermined time interval, the recording information of the voice / posture recording means, and the utterance target person specifying means Result Wherein the speech flow detection means for detecting, and display means for displaying the display information corresponding to the detection result of the speech flow detecting means, further comprising a based on.

[0017]

According to the present invention having the above-described structure, not only the utterance sections of the utterer are sequentially arranged in chronological order, but also the specific utterance target person specifying means from the posture information of each utterer.
It is specified to whom the remark is. And
An utterance flow detecting means detects an utterance flow reflecting information on who the utterance is for, and a display means,
Display information indicating the flow of the remark is displayed on the display screen.

Thus, it is possible to easily detect a portion where the interaction between the utterances is high, and it is possible for the user to predict what kind of utterance has occurred just by looking at the display information. Therefore, it is very useful for searching for a necessary part, such as searching for an important part of a meeting.

[0019]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing an embodiment of a speech structure detecting and displaying apparatus according to the present invention. The embodiment described below is a case where the present invention is applied to a meeting information recording / reproducing apparatus for a face-to-face meeting with three or more people. Before describing the embodiment of the utterance structure detection display device according to the present invention, an outline of a face-to-face conference of this example will be described with reference to FIG.

Each of the conference participants 1 is equipped with a sensor 2 for detecting the line of sight and a three-dimensional magnetic sensor 3 for detecting the position and direction of the head. As the line-of-sight detection sensor 2 used in this example, a sensor using a corneal scleral reflection method was used.

In order to individually record the voice of each of the conference participants 1, a microphone 4 is assigned to each of the conference participants.

The information obtained from the sensor 2 for detecting the line of sight of each conference participant 1, the three-dimensional magnetic sensor 3 and the microphone 4 is input to a personal computer 5 for recording and reproducing conference information and digitized. Is recorded. Then, as software of the personal computer 5, a main part of the speech structure detection and display device according to the embodiment of the present invention is realized.

FIG. 1 is a block diagram showing a speech structure detecting and displaying apparatus according to this embodiment. The speech structure detection and display device according to this embodiment includes a speech information processing unit 11 that processes a speech signal from a microphone 4 of each conference participant, and a posture input device 6 including a sensor 2 and a magnetic sensor 3 for line-of-sight detection. Posture information processing unit 12 for processing posture information of the user, a voice / posture recording unit 13 for recording the processed voice information and posture information, and a user input unit 14 for receiving a user input of a conference participant or a searcher of the conference information. A control unit 15 for performing control based on a user input; a user input recording unit 16 for recording user input of a conference participant;
3, a chart creating section 17 for creating a speaker chart visually expressing the state of the statement, a display section 18 for displaying information such as user input and the speaker chart, and a playback for playing recorded voice. A voice output unit 20 for outputting a reproduced voice, a voice target person specifying unit 21 for specifying to whom the voice is directed, and a voice flow detecting unit 22 for detecting a voice flow.

In this example, the voice information processing section 11 digitizes voice information and detects a speech section for each conference participant. The digitization part is configured by connecting a so-called sound board to a personal computer. Detection of a speech section for each conference participant is performed by software processing of a personal computer. That is, digitized voice information is processed, and who spoke when and during a conference is detected by software processing.

As described above, in this example, the sound of each conference participant is collected by the individual microphone 4, and the correspondence between the microphone and each conference participant is recognized in advance. The information on the correspondence between the microphone and the conference participants is set by each conference participant or the like before the conference starts, and is recorded in the voice / posture recording unit 13.

In this example, if the audio signal level from a certain microphone is equal to or higher than a predetermined certain level L1 and it continues for a certain predetermined time Δt1 or more, the corresponding level It is considered that the conference participant has started the speech, and when the voice signal level is less than the predetermined level L2 for more than the predetermined time Δt2, it is considered that the speech has ended. Then, a speech section is detected.

FIG. 3 is a flowchart showing a speech section detection process for a speech signal from each microphone 4.
FIG. 4 is a conceptual diagram for explaining the comment section detection process. In this example, the utterance section detection process of FIG. 3 is started when a voice of level L1 or higher is detected. In the explanatory diagram of FIG. 4, L1 = L2. However, the level L1 and the level L2 may of course be different.

As shown in the flow chart of FIG. 3, first, when a sound of level L1 or higher is input from the microphone, the process proceeds to step 101, where the sound exceeds the predetermined threshold level L1 for a predetermined time Δt1 or longer. And monitor whether the voice is sustained. If not,
It is not regarded as a speech, and the speech section detection processing routine ends.

As shown in FIG. 4, at time T1, step 1
If it is determined that the condition of No. 01 is satisfied, step 10
2, the information of the current time T1 is obtained, the utterance start time ts is set to ts = T1-Δt1,
It is sent to the posture recording unit 13 and recorded.

Next, the routine proceeds to step 103, in which the audio signal level is kept for a predetermined time Δt2 or more.
It is monitored whether the level falls below a predetermined level L2.
As shown in FIG. 4, at time T2, when it is detected that the voice has dropped below the level L2 for a predetermined time Δt2 or more, the process proceeds to step 104, and the utterance end time te is changed to te = T2−Δt2. The information is sent to the voice / posture recording unit 13 and recorded.

FIG. 5 shows an example of the data structure of the statement status table TBL1, which is recorded information of the statement status of the voice / posture recording unit 13. The record of “utterance ID” is an identification number sequentially assigned to the detected utterance.
The record of "speaker" is the name of the conference participant whose speech was detected. If you have a separate conference participant table that describes all conference participants and their identification information,
This “speaker” record may be participant identification information.

"Speech start time" and "speech end time"
Record, the utterance start time ts and the utterance end time te calculated by the above-described utterance section detection processing are recorded. The last record of “speaker” is a record of each utterance, which is specified by the utterer target specifying unit 21 described later.

Next, the processing of the posture information processing section 12 will be described. The posture information processing unit 12 detects from the output of the line-of-sight detection sensor 2 constituting the posture information input unit 6 and the output of the magnetic sensor 3 which person the speaker pays attention to and speaks.

The line-of-sight detection sensor 2 can detect the line-of-sight direction in the head coordinate system of the user wearing the sensor. In this example, the line-of-sight detection sensor 2 detects the line of sight of both eyes, and uses the line of sight of both eyes to detect the point of view in the head coordinate system, that is, the viewpoint. And
The three-dimensional magnetic sensor 3 is used to convert this viewpoint position into a viewpoint position in an absolute coordinate system.

The three-dimensional magnetic sensor 3 is mounted on the head of a conference participant, whereby the direction of the head in the absolute space is determined. From the information of the three-dimensional magnetic sensor 3 and the viewpoint position information in the head coordinate system, the viewpoint position of each conference participant in the absolute space is obtained. Posture information processing unit 1
2 records the location information of the conference participants. In this embodiment, the latest head position information output from the three-dimensional magnetic sensor 3 of each conference participant is recorded as the position information.

FIG. 6 is an explanatory diagram of a method of obtaining a watch target person (the watch target person who is speaking is the comment target person) in this embodiment.

As shown in FIG. 6A, in this embodiment, the viewpoint position Pe of a conference participant A is determined by the position of the three-dimensional magnetic sensor 3 mounted on the head of another conference participant B. When it is within a predetermined range around Ps (within a sphere of radius R), it is interpreted that the conference participant A is watching the conference participant B.

The process of detecting the watching target for each conference participant is performed, for example, in a unit time cycle. In the voice / posture recording unit 13, information on the watch target person within each unit time for each participant is recorded as a watch target person table TBL2, for example, as shown in FIG. The record of “time” in the watch target person table TBL2 is identification information of each unit time, and in this example, is indicated by a sequential number. In the example of FIG. 7, for example, it is recorded that the conference participant A gazes at the conference participant B at time 1 and time 2.

In the example described above, since the line of sight is detected by both eyes, the viewpoint can be obtained. However, a method is also conceivable in which the gaze target is approximately detected by using the line of sight of one eye and the three-dimensional magnetic sensor 3 or only the three-dimensional magnetic sensor 3. However, in this case, it is not always possible to detect that the eye is gazing.

FIG. 6B is a schematic diagram for explaining a method of detecting a person to be watched in this case. In this case, the distance d between the line DR representing the line of sight or the head direction and the position Ps of the three-dimensional magnetic sensor 3 of the participant B is obtained, and the distance is determined as the position of the three-dimensional magnetic sensor 3 of the participant B. If it is within a predetermined distance R from Ps, it is interpreted that the user is watching.

As described above, the speech / posture recording unit 13 stores the speech status table TBL1 which is a record of the speech status.
And the gaze table TBL2, which is a record of the posture status, is recorded, and audio signals from all the microphones 4 are recorded as conference information. The audio information is recorded in a normal audio format such as provided on a personal computer or a workstation.

The display section 18 has a display screen composed of, for example, a CRT monitor or a liquid crystal monitor. In this example, the display section 18 is configured to also serve as a pen / tablet integrated input / output device.

In this example, the user input unit 14 is constituted by the pen / tablet integrated type input / output device.
The control unit 15 receives user input information from the user input unit 14, sends the information to the display unit 18, and causes the display unit 18 to display the information. Then, processing according to the received user input information is performed at the time of information recording (at the time of information storage) or at the time of information reproduction.

The user input information includes, in addition to handwriting or graphics (objects such as lines, rectangles, circles, etc.) from a pen (or mouse / trackball / touch panel, etc.), code information for character recognition of handwriting data, Code information from a keyboard may be used.

Editing information that the displayed user input information has been moved / copied / deleted, information that the page has been switched, information that the user has sat on a chair with a sensor, and a virtual eraser are used. The information may be such that the user input information is not displayed, such as information indicating that the user input information is displayed. In this case, a predetermined sign indicating the presence of the user input information is displayed on the display unit 18. That is,
Any input made by a user to a device having a calculation processing capability while an input audio signal is present corresponds to the user input information referred to in the present invention.

The control unit 15 also records in the user input information recording unit 16 the user input information input at the time of recording the audio information as the conference information and the input start time and end time.

At the time of reproduction, the control section 15 reads information recorded in the user input information recording section 16 in response to a user's instruction from the user input section 14 and transmits the information to the display section 18. I do. Further, as described later, when the speaker chart is displayed on the display screen of the display unit 18 and an instruction is input by the user, a time portion corresponding to the instructed portion can be reproduced. It is configured to be able to.

The display section 18 displays the user input information passed from the control section 15 on its display screen. In addition, as described later, the speaker chart created by the chart creating unit 17 is displayed on the display screen.

The user operates the display unit 1 as described later.
From a plurality of user input information displayed on the display screen of No. 8,
Any one of the user input information can be selected for reproducing the corresponding time-series information or instructing to create a chart. Further, in this system, by specifying a specific reproduction position, that is, a reproduction start point or a reproduction section, in the speaker chart displayed on the display unit 18, reproduction of audio information corresponding to the specified reproduction position is performed. Can be performed.

As a recording medium of the voice / posture recording unit 13 and the user input information recording unit 16, a semiconductor memory or a hard disk built in a personal computer is specifically used. Note that a recording medium such as an MO disk or a floppy disk may be used.

The display unit 17 and the audio output unit 20 are realized by a display and a speaker connected to a personal computer.

Next, the processing operations of the utterance target person specifying unit 21, the utterance flow detecting unit 22, and the chart creating unit 17 realized by the software of the personal computer will be described.
This will be described below.

First, the operation of the comment target person specifying unit 21 will be described. When the user inputs a command for specifying the target person via the user input unit 14, the command is input to the target user specifying unit 21 via the control unit 15. The operation will be described below.

Using the information of the watch target table TBL2 recorded in the voice / posture recording unit 13, the target person specifying unit 21 specifies the target person of each speaker. In the case of this example, when specifying the utterance target, a time section is specified for each utterance, and the information of the watching target table TBL2 as the posture status information in the specific section is referred to. Identify the target speaker. The specific section is hereinafter referred to as a target section D.

The utterance start time is ts, and the utterance end time is t
e, let D be a target section considered as valid posture information in the utterance, and consider a time length t1 from the utterance end time te in the direction of the utterance start time ts and a time length t2 after the utterance end time te. Basically, a section from time te-t1 to time te + t2 is set as a target section D with the utterance end time te as a reference time. However, there are various ways to determine this. The entire utterance may be set as the target section D, or a section having a fixed ratio in the latter half of the utterance section may be set as the target section D.

There are four possible ways of determining the target section D, Case 1 to Case 4, and each case will be described with reference to the schematic diagram of FIG.

First, in case 1 shown in FIG. 8 (A), the time that has elapsed from the utterance end time te by time t1 is
The time (ts) between the speech start time ts and the speech end time te
<(Te−t1)), and in the section from the utterance end time te to the time point te + t2, if there is no other utterance, the section from the time point te−t1 to the time point te + t2 is set as the target section D.

In case 2 shown in FIG. 8 (B), the time preceding the utterance end time te by the time t1 is the utterance start time t
This is before s (ts> (te−t1)), in the section from the utterance end time te to the time point te + t2, and there is no other utterance. In this case, the section from the utterance start time ts to the time point te + t2 is set as the target section D.

In case 3 shown in FIG. 8 (C), the time that has elapsed from the utterance end time te by the time t1 is the utterance start time t
Time between s and utterance end time te (ts <te−t1)
However, this is a case where someone else's speech is made at time tx in the section from the speech end time te to the time point te + t2. In this case 3, a section from the time point te-t1 to the time point tx is set as the target section D.

In case 4 shown in FIG. 8 (D), the time preceding the utterance end time te by the time t1 is the utterance start time t
ts> (te−t1) before s, and the time tx in the section from the speech end time te to the time te + t2
And when someone else speaks. This case 4
In the case of, the section from the speech start time ts to the time point tx is set as the target section D.

Next, FIG. 9 shows a flowchart of the processing operation of the utterance subject identification unit 21. Target person specifying unit 21
Is a command for specifying a person to be remarked from the user input unit 14,
Upon arrival through the control unit 15, the processing is started, and the process proceeds to step 201.

In step 201, the voice / posture recording unit 1
Focusing on the first utterance in the utterance situation table TBL1 recorded in No. 3 and its utterance start time ts and utterance end time t
Find e. Next, the process proceeds to step 202.

In step 202, regarding the remark,
As described with reference to FIG. 8 described above, the target section D is determined, and the utterance target person of the target section D is determined with reference to the watching target table TBL2 of the voice / posture recording unit 13.
The details of this specifying method will be described later.

Next, the routine proceeds to step 203, where it is checked whether or not the processing is the last utterance recorded in the utterance situation table TBL1. If so, the utterance subject identification process is terminated; otherwise, the process proceeds to step 204, where attention is paid to the next utterance, the utterance start time ts and the utterance end time te are obtained, and the process proceeds to step 202. .

Next, a description will be given of a method of obtaining the utterance target in this example in step 202. In this example, with reference to the gaze target table TBL2, when the user gazes at a participant continuously for 3 unit time or more in the target section D specified in step 202, the gaze target person Is specified as the person to speak.

This will be described with reference to the example shown in FIG. This FIG.
Is a gaze target table T of the voice / posture recording unit 13.
The posture situation of the participant A of BL2 is shown. Now, temporarily,
It is assumed that the target section D specified in step 202 is specified to include a unit time from time n to time n + 15.

As shown in FIG. 10, in the case of this example,
In the target section D, when the same participant appears three or more times in a row, the participant becomes the utterance target. FIG.
In the case of the example, the participant B and the participant D are each specified once as a speech target person, as shown by arrows in FIG. 10, from time n to time n + 15.

Next, a description will be given of a process of expressing the state of a comment as a chart and displaying the chart on the display unit 18.

When a chart creation command and a time section (start time Ts, end time Te) of a chart to be created are input from the user input unit 14, the control unit 15 sends the information to the chart creation unit 17. . Chart creator 17
Receives this, executes a chart creation process. FIG. 11 shows a flowchart of an example of the chart creation process of the chart creation unit 17.

That is, the chart creation section 17 sends a chart creation command and a time section (Ts, Ts) of the chart to be created.
When e) is received, the process proceeds to step 301. Step 3
In step 01, a basic chart as shown in FIG. 12 is created by visually referring to the statement status table TBL1, which is the statement record information of the voice / posture recording unit 13, and expressing the speech section of each participant on the time axis. I do.

In the basic chart of FIG. 12, a conference participant name is displayed in an area 31. Next to the conference participant name display area 32 is a speaker chart display area 32, in which the specified start time Ts
The speech section of each conference participant between and the end time Te is
It is represented by a rectangular bar 33.

The speaker chart display area 32 has a vertical and horizontal size, a location on the time axis, and an area 31 indicating information on the speaker.
Is stored in the control unit 15, and display information is generated with reference to the position. In this embodiment, the size of the chart (the length of the time axis) changes according to the length of the designated section.

When the basic chart is created as described above, the process proceeds to step 302, where the utterance flow detecting unit 22
The information of the time section (Ts, Te) and the flow detection command are sent. In accordance with this command, the flow of the utterance is detected by the utterance flow detection unit 22 as described later, and information on the detected flow of the utterance is sent.

Therefore, in step 303, it is determined whether or not the information on the flow of the utterance from the utterance flow detecting unit 22 has been received.
Based on the result received from the utterance flow detecting unit 22, when there is a utterance target for the utterer in each utterance section, a rectangular bar 33 of the utterance and a rectangular bar 33 of the next utterance of the utterance target are displayed. Then, on the chart, as shown in FIG. Then, the process proceeds to step 305, and the chart is displayed on the display screen of the display unit 18.

Next, the utterance flow detection processing operation of the utterance flow detection unit 22 started by the instruction issued in step 302 will be described with reference to the flowchart of FIG.

That is, when the information of the time section (Ts, Te) and the utterance flow detection command are input from the chart creating unit 17, the process proceeds to step 401, where the voice / posture recording unit 1
With reference to the utterance ID record in the utterance status table TBL1 of No. 3, a plurality of utterance IDs in the designated time section (Ts, Te) are obtained.

Next, proceeding to step 402, focusing on the first utterance of the plurality of utterance IDs obtained,
Proceed to 3. In step 403, the voice / posture recording unit 13
With reference to the record of the utterance target person in the utterance status table TBL1, it is checked whether or not the utterance target person for the utterance of the utterance ID of interest is the next utterer.

Then, if the utterance target of the utterance of the utterance ID of interest is the next utterer, and the utterance target of the utterance of the utterance of the utterance ID of interest is Since the two speakers can be considered to be watching each other when they are speakers of the noted statement ID, the name of the speaker of the noted statement ID, the speech start time and the speech end time, and the speaker of the next statement The name and the speech start time are stored in the buffer as one set of information. FIG. 15 shows the information stored in the buffer at this time.

Next, the process proceeds to step 404, where the utterance ID of the utterance of interest is set in the designated time section (Ts, Ts
It is checked whether the utterance ID is one immediately before the end of e), and if so, the process proceeds to step 405, where the information stored in the buffer is sent to the chart creating unit 17, and the process ends. Otherwise, the process proceeds to step 406, and focuses on the next statement ID. Then, the process returns to step 403, and the same processing as described above is repeated.

FIG. 13 shows an example of the speaker chart created as described above and displayed on the display unit 18. The example of FIG. 13 is a chart in a case where the following utterance situation occurs in a meeting. That is, after the conference participant A speaks, the conference participant B speaks to the conference participant C, and the conference participant C answers. After a while, conference participant B commented on it. Next, the conference participant A receives the conference participant D
Said something to. Conference participant D then said something.

As shown in FIG. 13, the conference participant B speaks to the conference participant C, and the conference participant C responds to the part or the conference, as shown in FIG. Participant A made a statement to conference participant D,
The interactive part, their remark bar 3
3 are connected by an arrow 34 in FIG. 13 to indicate that fact.

As described above, by displaying the relationship between the statements on the chart from the statement information and the posture information, the user can recognize not only the change of the statements but also the flow of each statement.

By recording the record of the utterance target person in the utterance status table in more detail, the interaction between utterance sections can be displayed in more detail.

FIG. 16 shows an example of the statement status table TBL3 in such a case. In FIG. 16, FIG.
The comment subject column is recorded in more detail than the comment status table TBL1. That is, FIG.
In the case of, as the record of the utterance target, the number of times the utterer of the utterance ID gazes (the number of times the utterance target becomes) and the gazing time for each of the conference participants are respectively Be recorded. The uttered person specifying unit 21 records the number of times the uttered person became the uttered person and the time for each conference participant.

That is, in the case of this example, the target speaker specifying unit 21 records, for each conference participant, the number of times that the target is the target speaker and the time of that. As a result, the degree of gazing of the speaker in the conference to other conference participants can be understood, and more detailed interactivity can be displayed on the speaker chart.

As in the statement status table TBL3 of this example, as the record of the speaker, the number of times the speaker with the corresponding speaker ID gazes (the number of times the speaker becomes the speaker) is checked for all the conference participants. When the gaze time is recorded, as shown in FIG. 17, in the speaker chart, based on the information on the time and the number of times each conference participant became the speaker, as shown in FIG. Connecting line 3
5, 36 and 37 can be displayed with different attributes.

For example, in the example of FIG. 17, the interactivity is determined in accordance with the information of the gaze time and the number of times, and the attributes such as the thick solid line 35, the thick broken line 36, and the thin solid line 37 are determined in the descending order of the interactivity. Is changed and displayed.

Note that the comment flow detection unit 22 is not used only for creating a chart. For example, a utterance flow detection command and a time interval (T
s, Te) and the names of the two speakers are input, the time during which the specified two conference participants have exchanged in the time section (Ts, Te) is the utterance flow detection unit 22. Output from

This output is output to the display unit 18 via the control unit 15. In this example, the buffer information in the above-described chart creation is output. This is an example in which the utterance flow detection unit 22 is one component in the search.

It is not possible to accurately know the time of the highly interactive exchange by merely focusing on the transition of the speaker. Scenes where two specific parties interact, such as who answered, can be accurately extracted.

In this example, since the display unit 18 is an input / output integrated display, it is possible to reproduce the audio information by directly specifying the displayed speech flow by the user. In this case, input coordinates according to the user input are sent from the display unit 18 to the control unit 15.

Since all the displayed information, such as chart information and displayed instructions, is managed by the control unit 15, the control unit 15 interprets the meaning of the input. For example, when the user specifies an arbitrary position on the displayed chart and presses a play button, the control unit 15
After converting the coordinates into time, a reproduction command and time are sent to the reproduction unit 19. The reproduction unit 19 reads the recorded audio signal of the audio / posture recording unit 13 for the designated time, and outputs the signal to the audio output unit 20.

As another embodiment, an example in which a section specifying unit 23 is provided as shown in FIG. 18 is shown.

In this example, the section specifying unit 23 specifies a time section corresponding to the output of the utterance flow detecting unit 22 from the time input once or the time section information. The specified section is used by the chart creating unit 17 and the reproducing unit 19.

Specifically, when an appropriate time interval is input by the user, the time or a time interval of a series of utterance flows including that time zone is specified and displayed as a speaker chart, or Can be played. FIG.
Fig. 9 shows a conceptual diagram. FIG. 20 shows a flowchart in the case of this embodiment.

That is, as shown in FIG.
Alternatively, when the time section (T0, T1) is input, the process proceeds to step 501, and as shown in FIG. 19, the first start time Ta of the speech flow before the time T or T0 is found. Then, the process proceeds to step 502, where the time T or T
After 1, the end time Tb at which the speech flow ends first is found.

Next, the routine proceeds to step 503, where the start time T
a and the end time Tb are output to the control unit 15. In this case, as shown in FIG. 19, the time section may be set to a section (Ta ′, Tb ′) in which the time section is slightly widened. In this case, the context becomes somewhat easier to understand as the time interval is extended.

The time input to the section identification unit 23 may be directly input by the user from the user input unit 14. Further, the following usage may be used.

That is, as shown in FIG.
When the user designates a specific instruction point 41 on the speaker chart displayed in 8 by the user input unit 14, the control unit 15
Indicates the user input recording unit 1 based on the time at the designated position.
Reference is made to the user input information and the input time recorded in No. 6 and the input time is input to the section identification unit 23.

The above embodiment is a case where the present invention is applied to a normal face-to-face conference as shown in FIG. 2, but the present invention is also applicable to a video conference.

FIG. 22 is an explanatory diagram of a conference situation when the present invention is applied to a video conference. In this example, the conference information recording personal computer 5 is connected to a terminal personal computer 62 in the room 60 of each conference participant 61 via the network 50.

On the screen of the display 63 of the personal computer 62 of each conference participant 61, the screens of the other conference participants are displayed in a multi-window format. Above the terminal personal computer 62, a sensor constituting the visual line detection unit 64 is installed. The gaze detection unit 64 detects which conference participant the conference participant 61 is watching on the screen. The posture information detected by the line-of-sight detection unit 64 is transmitted to the conference information recording personal computer 5 through the network 50.
Sent to

In the case of this example, the video of each conference participant and its voice are acquired by the video camera 65 and transmitted to the conference information recording personal computer 5 through the network 50.

Then, in the conference information recording personal computer 5, in the same manner as described above, the statement status table TB
Comment status information is recorded as L1 or TBL3,
The posture status is recorded by the watching target person table TBL2. At the time of reproduction, a speaker chart as shown in FIG. 13 or FIG. 17 is displayed on the display screen, and is used to help search.

[0105]

As described above, according to the present invention, it is possible to detect a conversation flow which has not been realized in a conventional conference system or the like. Then, it is possible to reproduce audio information and display a chart using the audio information.

As a result, it is possible to easily detect a portion where the interaction between the utterances is high, and it is possible for the user to predict what kind of utterance has occurred just by looking at the display information. Therefore, it is very useful for searching for a necessary part, such as searching for an important part of a meeting.

For example, when there are two continuous remarks, it can be distinguished whether they are included in one flow or whether another conversation flow has started. For example, if it is another conversation flow, it is understood that it is not a conversation exchange such as a question and an answer included in one conversation flow. Anyone who has participated in the meeting will see this flow and will also help recall the meeting information.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment of a speech structure detection display device according to the present invention.

FIG. 2 is a diagram for explaining an outline of a conference to which the statement structure detection display device according to the present invention is applied;

FIG. 3 is a flowchart for explaining a method of detecting a speech section in the speech structure detection display device according to the embodiment of the present invention;

FIG. 4 is a diagram for explaining a method of detecting a speech section in the speech structure detection display device according to the embodiment of the present invention;

FIG. 5 is a diagram showing an example of recorded information of a speech status in the speech structure detection display device according to the embodiment of the present invention;

FIG. 6 is a diagram for explaining an example of a method of detecting a gaze state as an attitude of a speaker in the embodiment of the speech structure detection display device according to the present invention.

FIG. 7 is a diagram for describing an example of recorded information of a gaze state as a speaker's posture in the embodiment of the speech structure detection display device according to the present invention.

FIG. 8 is a diagram used to describe a method of specifying a person to be uttered in the embodiment of the utterance structure detection display device according to the present invention.

FIG. 9 is a flowchart for explaining an example of processing for specifying a person to be uttered in the embodiment of the utterance structure detection display device according to the present invention.

FIG. 10 is a diagram used to explain a method of specifying a person to be uttered in the embodiment of the utterance structure detection display device according to the present invention.

FIG. 11 is a flowchart illustrating an example of a speaker chart creation process in the embodiment of the speech structure detection display device according to the present invention.

FIG. 12 is a diagram showing an example of a basic speaker chart.

FIG. 13 is a diagram showing an example of a speaker chart in the embodiment of the speech structure detection display device according to the present invention.

FIG. 14 is a flowchart for explaining an example of a comment flow detection process in the embodiment of the comment structure detection display device according to the present invention.

FIG. 15 is a diagram used to explain an example of a comment flow detection process in the embodiment of the comment structure detection display device according to the present invention.

FIG. 16 is a diagram showing an example of recorded information of a speech status in the speech structure detection display device according to the embodiment of the present invention;

FIG. 17 is a diagram showing an example of a speaker chart in the embodiment of the speech structure detection display device according to the present invention.

FIG. 18 is a block diagram showing another embodiment of the speech structure detection / display apparatus according to the present invention.

FIG. 19 is a diagram used for describing the operation of another embodiment of the utterance structure detection display device according to the present invention.

FIG. 20 is a flowchart used to describe the operation of another embodiment of the utterance structure detection display device according to the present invention.

FIG. 21 is a diagram used for describing the operation of another embodiment of the utterance structure detection display device according to the present invention.

FIG. 22 is a diagram for explaining an outline of another example of a conference to which the speech structure detection display device according to the present invention is applied.

[Explanation of symbols]

 2 gaze detection sensor 3 three-dimensional magnetic sensor 4 microphone 5 conference recording personal computer 11 voice information processing unit 12 attitude information processing unit 13 voice / posture recording unit 14 user input unit 15 control unit 16 user input information recording unit 17 chart creation unit Reference Signs List 18 display unit 19 reproduction unit 20 voice output unit 21 uttered person identification unit 22 utterance flow detection unit 23 section identification unit

Claims (4)

[Claims] 1. A speech input means for collecting a speech of a speaker, a speech section detection means for detecting a speech section of each speaker from a speech signal from the speech input means, A posture detecting means for detecting a posture, recording voice information from the voice input means, information of a speech section for each speaker detected by the speech section detecting means, and a speech detected by the posture detecting means. The posture of each person, a voice / posture recording means for recording in association with each other, based on information recorded in the voice / posture recording means,
Remark target person specifying means for specifying who the remark is to, a flow of the remark in a predetermined time interval, based on the record information of the voice / posture recording means and the result of the remark target person specifying means A utterance structure detection and display device, comprising: utterance flow detection means for detecting; and display means for displaying display information according to a detection result of the utterance flow detection means. 2. The utterance structure detection display device according to claim 1, wherein at least each of the utterance sections of the predetermined time section detected by the utterance flow detecting means and a mutual relation of the utterance sections are: An utterance structure detection display device comprising a chart creator for creating a utterance chart to be displayed on a display. 3. The utterance structure detection display device according to claim 1, further comprising a section specifying means for specifying the predetermined time section. 4. The apparatus according to claim 1, further comprising: user input information recording means for recording user input information input in response to said utterance and the input time. Speech structure detection display device.