JP2002344915A - Communication grasping device, and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display in real time a present conversation and conversations traced back to the past by taking the contents and status of the conversations into account so as to smoothly and efficiently progress a conference. SOLUTION: The communication grasping method includes a steps of converting conversation information such as entered voice and finger language into characters, interpreting utterance history information including utterance contents and utterance states based on the present and past conversations from entered conversation information, attaching the utterance history information to the converted characters while keeping explicit performance of a cross- reference between utterers and the utterance history information, and displaying characters with the utterance history information attached thereto as a consecutive character string according to time serial transition in the order of reception.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention can be used, for example, in the field of a communication situation visual grasping device, and converts a conversation in a conference or the like into characters in consideration of current and past contents and situations, and displays the characters. The present invention relates to a communication comprehension device and a communication comprehension device capable of performing the same.

[0002]

2. Description of the Related Art Conventionally, conferences and meetings that are gathered in one place, and TVs through communication lines with distant places.
In communication at meetings, conversations are conducted using voice as a medium while face to face. In this case, a recording tape or the like is used to record the conversation content.

[0003]

However, there is no method of displaying a current conversation or a conversation that has been traced in the past in real time in consideration of the content and situation of the conversation in a conventional communication environment. .

[0004] For this reason, it is difficult to refer back to the contents of discussions in the past, and it is difficult to grasp the development of the discussions, it is difficult to cite, the same contents are repeatedly explained, May be missed.

[0005] In addition, since the existing TV conference room system is intended to be visually closer to a face-to-face conference, transmission of voices and face images is mainly performed, and the content and situation of the conversation as described above are also considered. It is not something that is displayed in real time with the addition.

On the other hand, in order to solve such a problem, it is considered possible to display the recognition result of the conversation content of the conference on a display by voice recognition, but it is possible to display face-to-face communication simply by displaying the voice recognition result. It is important to match the line of sight and to prevent natural movement of the line of sight.

[0007] Furthermore, simply using the text chat together with the voice conversation merely increases the output media of the participants, disperses the concentration, and does not solve the problem.

Accordingly, an object of the present invention is to display a current conversation or a conversation that has been traced in the past in real time in consideration of the content and situation of the conversation, thereby enabling a smooth and efficient meeting. , A communication grasping device, and a method thereof.

[0009]

SUMMARY OF THE INVENTION The present invention relates to an apparatus for recognizing speech information of a speaker and displaying the speech information in character form, comprising: input means for inputting the speech information; Conversion means for converting, from the input conversation information, decoding means for decoding the statement history information including the statement content and the statement status based on the current and past conversations, the correspondence between the speaker and the statement history information An adding unit for adding the utterance history information to the converted character while maintaining the clarity of the relationship, and displaying the character to which the utterance history information is added as a continuous character string according to a time-series transition in an input order. And a display control means for configuring the communication comprehension device.

Here, as the statement history information, the contents of the statement, the amount of the statement, the amount of the statement, the activity of the statement,
And a speech situation including the speech frequency.

The contents of the latest remark may be always displayed in the vicinity of the speaker, and the contents of the past remarks may be gradually moved over time while maintaining the context.

The content of the utterance of the speaker may be displayed with the character spacing determined so as not to overlap with the content of the utterance of the next speaker, and may be displayed with the character spacing becoming narrower and overlapping as the content of the past utterance increases.

The contents of the latest remark may be displayed quickly, and the contents of past remarks may be displayed slowly as it goes back to the past, and may stop at a certain position and disappear.

The character may be displayed in the form of a cloud in the vicinity of the speaker.

In the cloud-like form of the character, the shape of the cloud may be changed as a whole according to the utterance situation. The present invention relates to a method for recognizing a speaker's conversation information and displaying it in characters, wherein the step of inputting conversation information, the step of converting the inputted conversation information into characters, and the step of From the information, the step of decoding the statement history information including the statement content and the statement status based on the current and past conversations, and the conversion is performed while maintaining the clarity of the correspondence between the speaker and the statement history information. A communication grasp method by providing a step of adding the utterance history information to the set characters and a step of displaying the characters to which the utterance history information is added as a continuous character string according to a chronological transition in an input order. provide.

The present invention is a program for controlling the display of a speaker by converting the conversation information into characters. The program is recorded on a computer-readable recording medium, and includes conversation information including voice or sign language. , Converting the input conversation information into characters, and decoding, from the input conversation information, utterance history information including utterance contents and utterance states based on current and past conversations Adding the utterance history information to the converted character while maintaining the clarity of the correspondence between the utterer and the utterance history information; and And displaying the characters in a sequence according to the sequential transition in the order of input.
Provide a communication grasp control program.

According to the present invention, there is provided a medium recording a program for controlling the display of a speaker by converting the conversation information into characters by a computer. The control program causes the computer to input conversation information including voice or sign language. Converting the input conversation information into characters, decoding the statement history information including the statement contents and the statement status based on current and past conversations from the entered conversation information, and While maintaining the clarity of the correspondence relationship with the information, the statement history information is added to the converted character,
The characters to which the utterance history information is added are displayed in the order of input as a continuous character string according to a chronological transition.Here, the utterance content is determined so that the character interval does not overlap with the utterance content of an adjacent speaker. The characters are narrowed down and overlapped as the content of the past remarks overlaps, and slowly displayed as it goes back in the past, stops at a certain position and disappears, and makes the characters cloudy, In the vicinity, a medium in which a communication grasp control program is recorded is provided by displaying while changing the shape of the cloud as a whole according to the utterance situation.

[0018]

Embodiments of the present invention will be described below in detail with reference to the drawings. [First Example] FIGS. 1 to 4 show a first embodiment of the present invention.
It will be described based on. (Outline) First, an outline of the present invention will be described.

In implementing the present invention, the following three points:
Consider.

[0020] Secondary use of communication contents is necessary. For this purpose, voice is transcribed so that secondary use can be performed by visual information.

At this time, since it is important to use visuals that do not obstruct the line of sight of the dialogue, it is realized by displaying characters in a cloud shape near the speaker.

In order to grasp not only the contents of the communication but also the state of the communication, it is necessary to clarify the amount of utterance and the state, and the expression is made using the frequency and density of cloud display of characters.

Therefore, in the present invention, the following method is used.

Communication means that are not visualized (chronologically) in a recorded manner (for example, voice conversation,
Regarding the contents of past communication by sign language): 1) Convert the communication contents into characters by using a character conversion device. However, the time during which communication is interrupted (for example, the time during which no speech is made in conversation) is converted into a character as a blank space.

2) The transition of the contents of the communication to the displayable range in a state where the contents of the communication and the relationship between the parties are clearly maintained in the vicinity of the party communicating the characters using the communication means. Is displayed as a continuous character string.

Here, the display form is: 3) Characterized communication contents (including blanks) are sequentially displayed in a displayable range so as to maintain a chronological order.

4) A character other than the blank portion to be displayed is provided with a background effect around the font so as to emphasize the presence of the character.

5) With the passage of time, the display part of the contents of the past communication is gradually moved while maintaining the context, and the latest visualized contents of the communication are always displayed in the vicinity of the party (upper part or lower part). At the bottom).

6) The blank part (the time when there is no speech) is made transparent (nothing is displayed) or inconspicuous display (at least no characters).

(Communication Grasp Processing) FIG.
1 shows an example of a communication comprehension process according to the present invention. In this example, based on the following steps, a process of recognizing the conversation information of the speaker, converting it into characters, and displaying it is performed.

In step S1, conversation information is input.

In step S2, the input conversation information is converted into characters.

In step S3, from the input conversation information, utterance history information including utterance contents and utterance states based on current and past conversations is decoded.

In step S4, the statement history information is added to the converted character while maintaining the clarity of the correspondence between the speaker and the statement history information.

In step S5, the characters to which the utterance history information is added are displayed in the input order as a continuous character string according to a time-series transition.

Based on the above points, a specific example will be described below.

(Specific Example) In this example, an example in which voice is used as a communication means and a face-to-face oral meeting is assumed as a communication example will be described.

(Device Configuration) FIG. 4 shows a configuration example of a characterizing device as a communication grasping device.

The character conversion device 1 includes a voice input device 10 as an information input device, a voice recognition device 20, a display device 3
It is roughly divided into 0.

In this case, the voice input device 10 is composed of one microphone that collects voice as conversation information. The speech recognition device 20 includes a speech recognition unit 21 including a character conversion unit 22, a history decoding unit 23, and an addition unit 24. The display device 30 includes a display control unit 31 and a display screen 33 (ie, a projection screen) above each individual.
Are projected.

Here, the configuration of each unit according to the present invention is as follows.

The character converter 22 has a function of converting the input conversation information into characters.

The history decoding unit 23 converts the input speech
It has a function of decoding utterance history information including utterance content and utterance status based on current and past conversations. In this case, the statement content is information that is the latest and goes back to the past.
The speech status is information including the speech volume, the activity of the speech, and the frequency of the speech.

The adding section 24 has a function of adding the statement history information to the converted character while maintaining the clarity of the correspondence between the speaker and the statement history information.

The display control section 31 has a function of controlling the display of the character to which the utterance history information is added, as a character string connected according to a chronological transition in the input order, that is, in the recognition order.

In this example, the three devices 10, 20,
Although the configuration is divided into 30, the configuration is not limited to this, and it may be configured integrally in the apparatus. Further, the conversation information is not limited to voice, but may be other information such as sign language. In the case of sign language, a function of recognizing the movement of the hand and converting it into a character is required.

(Characterization processing) Next, the operation of the present apparatus 1 will be described.

FIG. 3A shows a scene of a face-to-face oral conference using the present apparatus 1.

Each voice A to C by speakers 101 to 103
Is input to the voice input device 10 and analyzed by the voice recognition device 20. Thereafter, the display device 30 outputs and displays the time-series character strings (including the present and the past) 201 to 203 corresponding to the speakers 101 to 103 in a cloud shape.

Hereinafter, the characterizing process will be described in detail.

Conference participants (here, speakers 101 to 10)
3) speaks through the voice input device (microphone) 10. The voice input by this is input to the voice recognition device 22.
Is converted to characters through

Although the number of microphones 10 placed on the table is one, if it is difficult to identify each individual from among a large number of voices, each individual should wear the microphone 10 separately. Alternatively, a process for learning the voice recognition of each individual only for the first time may be added. Further, in this example, a microphone is used as the voice input device 10, but the present invention is not limited to this, and the voice recognition unit 21 is provided with a headset 50 (see FIG. , A microphone 51 and a headphone 52).

The character A of the speaker 101 is immediately analyzed by the voice recognition device 20, and then the display device 3
0 is displayed on the display screen 33. The display screen 33 is not limited to the projection screen 33 projected by the projector 32 above each individual. For example, the display screen of each individual, a common screen that can be viewed from all participants, or a conference The wall or the like of the room where the operation is performed can be used.

(Display Example) FIG. 2 shows the statement A of the speaker 101.
Indicates the contents displayed as a character string 201 on the display screen 33 in the form of characters.

A character string 20 in which the remark A is visualized in a character shape
1 is displayed in a direction above the speaker 101, such as near the ceiling, with a determined width so as not to overlap with the statement display of the next speaker. In this case, the displayed character string 201 is fast at first, slows down as it goes up, stops at a certain position, and disappears. While displaying characters, they move upward at a relatively high speed.

In such a display, the character string 201 in which the utterance A is converted into characters is displayed sequentially (in the order of recognition, that is, in the order of conversion into characters) from the upper part of the utterer 101, and the lower part of the display area. It is displayed horizontally from the edge.

As the character string 201 displaying the message A accumulates, the display area on the display screen 33 is scrolled, and the latest message is displayed at the bottom.

Then, as the time elapses, the amount of remarks is accumulated, and the immediately preceding remark A is always visible.
In order from the top (oldest utterance), the character display color becomes light and inconspicuous or disappears, or the character size is reduced, thereby securing the entire display area.

The scrolling speed of the characters is not necessarily the same for all the participants because the amount and speed of each person's utterance vary, but the oldest utterance characters are used so that the utterance characters cannot be displayed completely. Adjustment is made by reducing the size of the utterance or by increasing the scroll speed for a person whose utterance amount and speed are large.

The displayed character string 201 has a translucent background to each character font, and looks like a cloud when the entire remark is made. Silence (Since there is no remark,
At the time that is not converted to characters, a transparent part without a background (no clouds) is displayed (that is, a space is displayed according to the blank time), and it is visually distinguished from the part (steam cloud part) where the utterance described in the preceding paragraph is present You can do it (that is, if you don't speak, there's no cloud above your head). As a result, there is a cloud around the displayed characters and there is no cloud in the silent part, so that the shape of the whole cloud can be seen to change according to the amount of speech and the frequency.

When the same word is repeatedly spoken in a short period of time, an emphasis effect such as bolding of the displayed characters, display of the characters faintly, change of the character color, and the like is given. You may.

(Summary) The advantages of the characterizing process are summarized as described above.

1) By displaying the contents of a statement above (or below) the individual speaker or in the vicinity of the person himself / herself, it is possible to clearly visualize who has spoken and how much.

2) The content of a total utterance within a certain period of time in the past or the total utterance content is displayed, and the difference between who utters and how much can be visualized quantitatively.

3) The history and frequency information of the comment can be visually referenced.

4) When the displayed image is not shared, for example, when there is only one person at one place, the past utterances hidden by the self / other / speaker can be scrolled and referenced as necessary.

5) The total time or the amount of remarks within a certain period of time in the past is displayed at a position that does not make the user feel uncomfortable with the conversation space. You can refer to and get information that makes the next statement more effective. For example, it is possible to promptly make a statement when the content of the statement can be judged calmly or when the distribution of persons who have not made a statement is clear.

[Second Example] A second embodiment of the present invention will be described with reference to FIG. The description of the same parts as those in the first example is omitted, and the same reference numerals are given.

FIG. 5 shows an example in which a remote conference using a TV screen or the like is assumed. The configuration of the character encoding device 1 is the same as that of FIG. In this example, a headset for voice recognition may be used.

The uttering devices 101 to 10
3 are converted into characters and displayed on the display screen 33 as cloud character strings 201 to 203. These speakers 101 to 103 and the displayed character strings 201 to 203 are photographed by the TV camera 400 and sent to a distant partner.

On the other hand, on the TV screen 60, the utterance contents of the utterers 301 and 302, which are distant partners, are displayed as cloud character strings 401 and 402.

As described above, by using the character encoding device 1 for the remote conference, even if the image of the remote participant is on the TV screen 60, the display of the utterance content of the person concerned is close (for example, the upper part of the participant). ) By displaying in the space, a part of the content of the other party's remarks will be present on this side, the sense of distance that can not be wiped out in a TV conference can be reduced, and a sense of intimacy can be felt in some way, It can be done smoothly.

When the speakers 101 to 103 use different languages for conversation, such as Japanese and the speakers 301 and 302 use English, a translating function is further provided in the characterizing device 1. It is possible to hold meetings more smoothly.

[0074]

As described above, according to the present invention,
Conversation information such as input voice and sign language is converted into characters, and from input conversation information, utterance history information including utterance contents and utterance status based on current and past conversations is decoded, and utterer and utterance history information While maintaining the clarity of the correspondence relationship with, the utterance history information is added to the converted character, and the characters to which the utterance history information is added are displayed in the input order as a continuous character string according to a chronological transition. Communication content that has identified itself and others, and
It is possible to visually present past communication contents that were difficult to refer to due to unrecorded data, together with indices such as the amount of speech of the communication, the activity and frequency of speech, which could not be visually recognized conventionally, and , Communication can be multi-layered and denser, and current and past conversations can be displayed in real time, taking into account the content and situation of the conversation, and consequently meetings can be held smoothly and efficiently .

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating a communication grasp process according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram showing the contents of a cloud-like character string displayed on a display screen.

FIG. 3A is a perspective view showing a scene of a face-to-face oral meeting,
(B) is a perspective view showing how to speak using a headset.

FIG. 4 is a block diagram illustrating a configuration of a character encoding device according to the present invention.

FIG. 5 is an explanatory diagram showing a remote conference which is a second embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Characterization device (communication grasping device) 10 Information input device 20 Voice recognition device 21 Voice recognition unit 22 Character conversion unit 23 History decoding unit 24 Addition unit 30 Display device 31 Display control unit 32 Projector 33 Display screen 60 TV screen 101 to 103 Speaker 201-203 Character string 301-302 Speaker 400 TV camera

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI theme coat ゛ (reference) G10L 15/00 G10L 3/00 551G 15/22 561C F term (reference) 5B091 AA11 BA19 CA02 CA14 CB12 5C064 AA02 AA06 AB06 AC02 AC06 AC08 AC09 AC12 AC14 AC18 5D015 KK02 5E501 AA01 AB20 AC34 BA03 CA02 CB15 EA21 FA13 FB32

Claims (17)

[Claims] 1. An apparatus for recognizing speech information of a speaker and displaying the speech information in a character form, comprising: input means for inputting conversation information; conversion means for converting the inputted conversation information into characters; Decoding means for decoding utterance history information including utterance content and utterance status based on current and past conversations from the obtained conversation information, and a state in which the correspondence relationship between the utterer and the utterance history information is maintained. And adding display means for adding the utterance history information to the converted character, and display control means for displaying the character added with the utterance history information as a continuous character string in a time-series transition in the order of input. Communication grasping device characterized by that. 2. The communication according to claim 1, wherein the utterance history information includes utterance contents that are traced to the latest and the past, and utterance status including utterance volume, utterance activity, and utterance frequency. Holding device. 3. The display control means always displays the latest utterance content in the vicinity of the utterer, and gradually moves the display position of the past utterance content while maintaining the context with the passage of time. The communication comprehension device according to claim 1, wherein the communication comprehension is performed. 4. The display control means, wherein the content of the utterance of the utterer is determined and displayed so that the utterance content of the utterer does not overlap with the utterance content of an adjacent utterer. The communication comprehension device according to any one of claims 1 to 3, wherein the communication grasp device is displayed in an overlapping manner. 5. The display control means according to claim 1, wherein the latest utterance content is displayed quickly, the past utterance content is displayed slowly as it goes back to the past, and stops at a certain position and disappears. 5. The communication grasp device according to any one of 1 to 4. 6. The communication comprehension device according to claim 1, wherein the display control means displays the characters in the form of a cloud in the vicinity of the speaker. 7. The communication comprehension device according to claim 6, wherein the cloud-like form of the character has a cloud-like shape that changes as a whole according to the speech situation. 8. A method of recognizing conversation information of a speaker and displaying the conversation information in characters, comprising: inputting conversation information; converting the input conversation information into characters; From the conversation information, the step of decoding the statement history information including the statement content and statement status based on the current and past conversations, and, while maintaining the clarity of the correspondence between the speaker and the statement history information, A communication comprising: a step of adding the comment history information to the converted characters; and a step of displaying the characters to which the comment history information is added as a continuous character string according to a chronological transition in an input order. How to figure out. 9. The communication according to claim 8, wherein the utterance history information includes utterance contents retroactively and retroactively, and utterance status including utterance amount, utterance activity, and utterance frequency. How to figure out. 10. The latest utterance content is always displayed in the vicinity of the utterer, and the past utterance content gradually moves the display position while maintaining the context with the passage of time. The communication grasp method according to claim 8 or 9. 11. The statement content of said speaker is displayed with character spacing determined so as not to overlap with the comment content of an adjacent speaker, and the character spacing is narrowly overlapped and displayed as the content of the past statement becomes larger. The communication grasping method according to any one of claims 8 to 10, wherein: 12. The content of the latest statement is displayed quickly, and the content of the past statement is slowly displayed as it goes back to the past, and stops at a certain position and disappears. Communication grasp method described in. 13. The communication according to claim 8, wherein the character is displayed in the form of a cloud and the shape of the cloud is changed in the vicinity of the speaker while changing the shape of the cloud according to the state of the utterance. How to figure out. 14. A program for characterizing and controlling the display of conversation information of a speaker, the program being recorded on a computer-readable recording medium, and inputting conversation information including voice or sign language. And the step of converting the input conversation information to characters, and, from the input conversation information, decoding statement history information including statement content and statement status based on current and past conversations, Adding the utterance history information to the converted character while maintaining the clarity of the correspondence between the utterer and the utterance history information; and And displaying the characters as a continuous string according to the transition in the order of input. 15. The utterance content is displayed with a character interval determined so as not to overlap with the utterance content of an adjacent speaker, and the character interval is narrowed and overlapped as the past utterance content is increased. The communication grasp control program according to claim 14, wherein the communication grasp control program is displayed slowly as it goes back, stops at a certain position, and disappears. 16. The character is displayed in the form of a cloud, and is displayed in the vicinity of the speaker while changing the shape of the cloud as a whole according to the state of the utterance.
Communication grasp control program described. 17. A medium storing a program for controlling display of a speaker by converting the conversation information into characters by a computer, wherein the control program causes the computer to input conversation information including voice or sign language. The input conversation information is converted into characters, and from the input conversation information, the statement history information including the statement contents and the statement status based on the current and past conversations is decoded, and the speaker and the statement history information are interpreted. While maintaining the clarity of the correspondence relationship, the statement history information is added to the converted character, and the characters to which the statement history information is added are displayed in the input order as a continuous character string according to a chronological transition. Here, the utterance content is displayed with the character spacing determined so as not to overlap with the utterance content of the next speaker, and the character spacing is determined as the past utterance content becomes higher. While narrowing and overlapping, it is displayed slowly as it goes back in the past, stops at a certain position and disappears, makes the character cloud-like, and in the vicinity of the speaker, the entire shape of the cloud according to the said speech situation A medium storing a communication grasp control program characterized in that the program is displayed while being dynamically changed.