JP2003323628A - Device and program for video identifying speaker and method of displaying video identifying speaker - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for a video identifying a speaker which facilitates identification of the speaker and understanding of the contents of conversation and provides a video having a sense of presence while dispensing with large-scaled equipment when making the conversation via a network, and to provide a program for the video identifying the speaker and a method of displaying the video identifying the speaker. <P>SOLUTION: This device for the video identifying the speaker is configured of: a data inputting part 2 for inputting conversation data constituted of text data and voice data; a data analyzing/extracting part 4 for analyzing the conversation data, and for extracting identification data and structural data indicating the structural characteristics of the conversation data, and for extracting contents analytic data indicating the contents characteristics of the conversation data; a video generating/ displaying part for generating a video including the CG characters of the speaker based on the structural data, the contents analytic data, and the identification data, and for displaying the video including the CG characters along the configuration of interactive program data prepared beforehand on a display screen 7 of a terminal; and a storage part 4 for storing the respective data. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a speaker identification video apparatus and its program, and a speaker, which are used when a plurality of speakers perform a synchronous or asynchronous conversation using text data or voice data over a network. The present invention relates to a specific video display method.

[0002]

2. Description of the Related Art Conventionally, it is known that information can be exchanged synchronously or asynchronously between a plurality of speakers using terminals connected to a network. For example, in this synchronous or asynchronous information exchange (hereinafter, simply referred to as conversation), a remote conference system (when synchronized) in which a plurality of speakers at a remote place talk in real time, or a plurality of unspecified Chat can be performed in real time at the conversation places (chat rooms) established on servers scattered on the network (when synchronized), or by posting on the electronic bulletin board held on the server for conversation. Use the electronic bulletin board system (for asynchronous) and the mailing list of the mail software installed on the terminal to send emails to multiple people at once and receive reply emails from the multiple people to have conversations. (In case of asynchronous) etc.

However, in a conversation between a plurality of speakers using a terminal connected to a network, most of the data transmitted and received are only text data and voice data. Therefore, when trying to smoothly recognize who is speaking among the speakers participating in the conversation, the amount of information is small only with text data or voice data, and it is difficult to recognize (the speaker cannot be identified. In some cases, it is difficult for the speakers to understand the content of their remarks, and the participants participating in the conversation tend to lack a sense of presence and lack of concentration. In order to solve these problems, a technique of adding video to the text data and audio data has been realized.

For example, as a conversation between a plurality of speakers in a state where a video is added to text data and voice data, a live image which is an image of a participant actually participating in the conversation, and the participation. A video conferencing system that sends and receives voices spoken by people, a simple remote conference system that sends and receives moving images as compressed images or still images as a single image (still image), creates video avatars from actual moving images, and creates them in a virtual space. The communication system to arrange is mentioned. In addition, a virtual whiteboard mainly intended to share information (conversation information) that is the result of a conversation by multiple people, a proxy conference using a robot with artificial intelligence built in as a proxy for the participants participating in the conversation, Prior art exists.

Further, regarding the conventional technique for expressing an image added to text data or voice data, in the terminal used by the participants who participate in the conversation, the image is displayed depending on the direction of the line of sight toward the display screen of the terminal. The eye-gaze matching display (which is based on eye-gaze matching technology) with varying numbers and text data input from multiple terminals are used as keywords to match the keywords, and the matching result is added to the text data and voice data. There is image processing by keyword matching that is reflected in the video.

[0006]

However, the following problems have been encountered in conversations in which video is added to text data or voice data using a conventional terminal connected to a network. Video conferencing systems that transmit and receive live video and audio, and simple remote conferencing systems that transmit and receive compressed images and still images, compared to conventional remote conferencing systems that transmit and receive text or audio data. The capacity is remarkably large. Especially, in the video conferencing system, since the images of the participants participating in the conference are transmitted and received as the moving images, privacy problems are likely to occur, and large equipment costs and cameras are required. There is a problem that it is necessary to prepare the specific equipment of.

Further, as an example of chat or the like, in general avatar chat or the like, a system in which text data input by a speaker from a terminal is conversed via a CG character in a virtual space which is the speaker's agent. There is also, but the screen configuration corresponding to the number of each CG character,
Since no consideration was given to the type of shot or the screen state such as the screen switching timing, it was difficult for the speakers participating in the conversation to recognize the screen state corresponding to the conversation. .

Therefore, it is difficult to identify the speaker, and it is difficult to understand the content of the conversation based on the flow before and after, so that there is little realism, and as a result, the concentration is likely to be lacked. There were problems such as being lost.

The present invention was devised in view of the above problems, and when a plurality of speakers talk using a terminal connected to a network without the need for large-scale equipment, (EN) Provided are a speaker-specific video device, a program thereof, and a speaker-specific video display method in which the speaker can be easily specified, the content of the conversation can be easily understood, and the image added to the conversation has a sense of realism. With the goal.

[0010]

In order to achieve the above-mentioned object, the present invention relates to a speaker specifying video device as described below. That is, the speaker identifying video device according to claim 1 uses a terminal connected to the network with text data or voice data to which identification data for identifying the speaker is added, and A speaker identification video device that facilitates identification of the speaker when a conversation is performed synchronously or asynchronously, and a data input unit for inputting conversation data by the text data or the voice data for conversation. , Analyzing the conversation data input to the data input unit to extract the identification data and the structural data indicating the structural features of the conversation data, and the content analysis data indicating the characteristic features of the conversation data. The data analysis extraction unit to be extracted, and the structural data and the content analysis data analyzed and extracted by the data analysis extraction unit. And a video including the CG character of the speaker is generated based on the identification data, and the video including the CG character is displayed on the display screen of the terminal according to the structure of the interactive program data prepared in advance. And a storage unit for storing the above-mentioned respective data.

With such a configuration, in the speaker specifying video device, first, the data input unit transmits the conversation data by the text data or the voice data to which the identification data for identifying the speaker is added. Input from.

When the conversation data to which the identification data is added is input, the data analysis / extraction section analyzes the conversation data to identify the speaker and the structural characteristics of the conversation data. For example, structural data such as the statement length data indicating the statement length of the speaker is extracted, and the content characteristics of the conversation data are extracted, for example, the speaker name quote indicating whether or not the speaker name is quoted. Content analysis data such as data is extracted and output to the video generation display unit. The identification data used here is an identification ID given by the sender, a nickname added by the speaker,
Alternatively, the physical I in the sender's network communication
D and so on. When the conversation data is voice data, the speaker is identified by comparing it with the voice data registered in advance (the speaker is identified by matching the voiceprints), and the identification based on the comparison result is performed. Data may be generated and added to the audio data.

Then, in the image generation / display section, based on the identification data, the structure data and the content analysis data,
A video including a CG character as a speaking agent of the speaker in the virtual space is generated, and the video is displayed on the display screen of the terminal that has input the conversation data. At this time, for example, the image may be created by setting the arrangement of the CG characters and the background image using TVML (TV Program Making Language). Then, the video including the CG character is stored in the storage unit and prepared in advance, and the screen configuration, screen transition, It will be configured according to the screen transition timing. Then, in this video generation / display unit, voice synthesis data obtained by performing voice synthesis based on the input conversation data is added to each CG character.

According to a second aspect of the present invention, in the speaker identification video device according to the first aspect, the data analysis / extraction section uses, as the structure data, at least the speech length of the speaker. Is provided with a conversation structure analysis and extraction unit that analyzes and extracts utterance length data indicating the utterance number and serial number data indicating the serial number in the utterance order of the utterance, and at least the citation of the utterance name is used as the content analysis data. Conversation content analysis / extraction means for analyzing and extracting speaker name citation data indicating presence / absence and utterance content citation data indicating presence / absence of utterance content by other speakers is provided.

With such a configuration, in the speaker-identifying video device, the data analysis / extraction section uses at least the statement length data indicating the statement length of the speaker and the statement of the speaker out of the structure data. The serial number data indicating the sequential serial numbers and the speaker data indicating which speaker is speaking are analyzed and extracted. And the content analysis and extraction means
Of the content analysis data, at least the speaker name citation data indicating whether or not the speaker name is quoted, and the utterance content citation data indicating whether or not the utterance content of another speaker is quoted are analyzed and extracted, Each data is transferred to the video generation / display section.

Further, the speaker specifying video device according to claim 3 is the speaker specifying video device according to claim 1 or 2, wherein the video generation / display section is a screen configuration of a video including the CG character. Deciding screen transition means, a screen transition means for transitioning the screen configuration based on the transition probability of the screen configuration of this screen structuring means and the speech length data of the speaker, and transition timing of this screen transition means And a screen transition timing means.

With such a configuration, the speaker-identifying video device uses the screen composing means, the screen transition means, and the screen transition timing means for the screen composition, the screen transition, and the screen transition timing from the interactive program data. A video including a CG character having a conversation is constructed.

The speaker-specific video device according to claim 4 is the speaker-specific video device according to any one of claims 1 to 3, wherein the data analysis / extraction section is based on the conversation data. Thus, the facial expression data analysis and extraction means for extracting facial expression data for changing the facial expression of the CG character is provided. With such a configuration, the facial expression data analysis and extraction unit of the data analysis unit can reflect the facial expression of the CG character in response to each utterance of each speaker.

The speaker-specific video device according to claim 5 is the speaker-specific video device according to any one of claims 1 to 4, wherein the data analysis / extraction unit is configured to perform the conversation data. Based on the above, the CG character is arranged at the apex position of the octagon including the camera line-of-sight position, and based on the content analysis data, the seating reference of the CG character is set at a position facing the camera line-of-sight position. The speaker placement analysis means for determining the points and sequentially determining the placement of the remaining CG characters to the left and right of the seating reference point is provided.

With such a configuration, when the speaker placement analysis means determines the placement of the CG characters based on the content analysis data when the CG characters are used as a video image, it is central to the conversation. It is possible to determine the CG character that plays a role as the seating reference position facing the camera line-of-sight position, and sequentially arrange the CG characters, such as the number of times of utterance, close to the seating reference position.

Further, the speaker specifying video program according to claim 6 uses the terminal connected to the network by the text data or voice data to which the identification data for identifying the speaker is added to make a plurality of statements. The computer is caused to function by each of the following means in order to generate a speaker specifying image that makes it easy to specify the speaker when the person talks synchronously or asynchronously.

That is, each of the means is a data input means for inputting the conversation data of the text data or the voice data for conversation, and the identification data and the identification data by analyzing the conversation data input to the data input means. Data analysis and extraction means for extracting structural data indicating the structural characteristics of the conversation data and extracting content analysis data indicating the content characteristics of the conversation data, analyzed and extracted by the data analysis and extraction unit. An image including the CG character of the speaker is generated based on the structure data, the content analysis data, and the identification data, and the image including the CG character is generated according to the configuration of the interactive program data prepared in advance. And image generation and display means for displaying on the display screen of the terminal.

With such a configuration, in the speaker specifying video program, when a plurality of speakers have a conversation, the data analysis and extraction means extracts the input conversation data from the conversation data.
The structure data, the identification data, and the content analysis data are analyzed and extracted so that the image including the CG character of the speaker can be generated and displayed by the image generation display. In addition,
This speaker specific video program may be stored anywhere (any location) on a terminal or server on the network.

Further, the speaker-specific video display method according to claim 7 uses a plurality of terminals connected to the network by text data or voice data to which identification data for identifying the speaker is added. A speaker identification video display method that facilitates the identification of the speaker when the speaker performs the conversation synchronously or asynchronously, and inputs the conversation data by the text data or the voice data for conversation. A step of analyzing the conversation data to extract structural data indicating the structural characteristics of the identification data and the conversation data, and extracting content analysis data indicating the characteristic characteristics of the conversation data; Based on the structure data, the content analysis data, and the identification data, an image including the CG character of the speaker is displayed. And it generates a video including the CG character, along the interaction of program data structure that is prepared in advance, was to include a step of displaying on the display screen of the terminal.

With such a configuration, in the speaker specific video display method, structural data indicating structural characteristics, identification data, and content analysis data indicating content are extracted from the input conversation data. Then, based on the structural data, content analysis data, and identification data,
It is possible to display the structure of the video including the CG character of the speaker according to the structure of the interactive program data prepared in advance.

[0026]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. (Structure of Speaker Identification Video Device) FIG. 1 is a block diagram showing a speaker identification video device. As shown in FIG. 1, the speaker identification video device 1 is connected to a plurality of terminals owned by the speaker via a network 6 (Internet or the like), and text data or text data is input via an input unit 8 of the terminal. A data transmitting / receiving unit 2 as a data input unit for receiving conversation data of voice data and inputting the conversation data;
The data analysis / extraction unit 3 that analyzes the conversation data from the speaker input to the data transmission / reception unit 2 and extracts predetermined data, and the data analysis / extraction unit 3 analyzes the data and extracts each speaker. The video generation / display section 4 for generating the CG character of No. 1 and displaying it on the video display section 7 which is the display screen of the terminal, and the storage section 5 for storing each data.

In the description of this embodiment, the speaker specifying video device 1 is used as a central mechanism and each terminal is used as a centralized control system, but the speaker specifying is performed instead of each terminal. There is no problem even as a distributed control type system including a plurality of video devices 1. So in this case,
The speaker identification video device 1 includes the video display unit 7 and the input unit 8.

First, the terminal will be described. The terminal is a PC, a mobile terminal (mobile computer, PDA, etc.), a mobile phone, or the like connected to the network, and this terminal includes a video display unit 7 and an input unit 8. The video display unit 7 displays the C transmitted from the speaker identification video device 1.
An image including a G character is displayed. The input unit 8 is for inputting voice data or text data of a speaker as conversation data, and here,
It is equipped with a keyboard or voice microphone.

When the data input from the input unit 8 of the terminal is voice data, it is determined by which speaker the voice data input by the voice extraction software provided in the terminal belongs. A speaker means specific image device in which a comparison means (not shown) or the like is used, identification data is created after being compared with voice data stored in advance, and the identification data is attached to the voice data. It may be configured to transmit the data to the No. 1 (for example, JP 2001-69436 A).

Next, the speaker identification video device 1 will be described. The data transmission / reception unit 2 receives the speaker identification data and conversation data (text data, voice data) sent from the input unit 8 of the terminal, and the data analysis / extraction unit 3 and the video generation / display unit 4 make a speech. Image data showing a conversation state based on an image including the CG character of the speaker, the speaker or the network (Internet etc.)
Is configured to be transmitted to a terminal device (PC or the like) connected to. The conversation data used here is the text data input from the keyboard,
It is audio data input from the audio microphone. The conversation data received by the data transmitting / receiving unit 2 is once stored in the storage unit 5.

The data analysis / extraction unit 3 includes the data transmission / reception unit 2
Conversation structure analysis and extraction means 3A for extracting the identification data of the speaker from the conversation data received in step S1 and analyzing and extracting the structure data and the identification data indicating the structural characteristics of the conversation data, and the conversation data in detail. Conversation content analysis and extraction means 3D for analyzing and extracting content analysis data showing various characteristics, and speaker placement analysis means 3H for analyzing and extracting placement data for determining the placement of the speaker on the image from the conversation data. , And a gesture analysis unit 3J for analyzing and extracting data for adding a gesture or emotional expression of the speaker's CG character from the conversation data.

The conversation structure analysis and extraction unit 3A separates the identification data added to the conversation data or analyzes which speaker is speaking from the conversation data to extract the identification data. 3a, serial number analysis means 3b that analyzes serial numbers in the speaking order of the speaker from the conversation data and extracts the serial numbers as serial number data, and analyzes and extracts the speech length data of the speaker that indicates the structural characteristics of the conversation data. The speech length analyzing means 3c is provided.

The speaker analysis means 3a is for analyzing and extracting identification data attached to or inherent in the conversation data. The identification data used here is the identification ID added by the application of the terminal device on the transmission side.
In addition, there is a nickname added to the conversation data by the speaker, or a physical ID in network communication of the system included in the terminal device on the transmitting side. Also,
When the conversation data is voice data, it is possible to add an identification number to make the identification data by detecting the degree of coincidence with the voice data of each speaker registered in advance (Japanese Patent Laid-Open No. 2001-2001). 69436).

The serial number analysis means 3b identifies serial number data as the speaking order of the speaker at fixed time intervals from the time series accumulation of conversation data, basically based on the order of the accumulation time. The serial number analyzing means 3b analyzes the arrangement of each speaker from the content of the statement as serial number data, and determines the speaker arrangement analyzing means 3H.
It may be determined in consideration of the arrangement data of.

When the conversation data is voice data, the utterance length analyzing means 3c analyzes (measures) the actual utterance time to extract (specify) the utterance length, and the conversation data is text. In the case of data, the speech length of the speaker calculated (analyzed) from the speech synthesis speed is extracted (specified).

The conversation content analysis / extraction means 3D analyzes and extracts from the conversation data the speaker name quotation data that is extracted and analyzed as the speaker name quotation data indicating the presence or absence of the quotation of another speaker such as the name and nickname of another speaker. The analysis unit 3d, the comment content quotation analysis unit 3e that analyzes and extracts the comment content quotation data that indicates whether or not the comment content of another speaker is quoted from the conversation data, and the facial expression of the CG character of the speaker from the conversation data. The facial expression data analyzing / extracting means 3f for analyzing and extracting the facial expression data for changing is provided.

The speaker placement analysis unit 3H sets the maximum number of speakers to 7 and arranges them in time series so that each speaker is placed at the vertex position of the octagon including the camera line-of-sight position. The position of each speaker is determined by analyzing the conversation data. If there are eight or more speakers, a predetermined moderator (set as a facilitator of the conversation, for example, the first speaker who speaks first) should act on behalf of another speaker. I have to.

Here, FIG. 5 shows an arrangement state according to the number of speakers in the virtual space generated by the image generation / display unit 4 based on the analysis result by the speaker arrangement analysis means 3H. FIG. 5: is a schematic diagram which shows the arrangement | positioning state from 2 persons to 7 persons planarly. In addition, when the number of speakers is an even number, when the position of the moderator is the seating reference point, it may be on the right side or the left side of the moderator.

In the speaker placement analysis means 3H, the speaker who is the center of the comment is set as the moderator (first speaker) based on the content of the comment, and the position of each speaker on the image is set. In this speaker arrangement analysis means 3H, when considering the arrangement of each speaker, the position of the moderator is set as the reference arrangement,
The conversation data arranged in chronological order is examined to determine the speaker arrangement order based on the following elements (A) to (C). Hereinafter, a speaker is simply referred to as a speaker, and a speaker who listens to the speaker's remarks or tries to respond to the speaker's remark is a speaker, and the speaker and the speaker are combined, Sometimes called performers (including the moderator).

That is, as the element (A), the speakers who continuously speak are arranged on the video screen so that they are closer to each other. As an element (B), a speaker who frequently speaks is arranged on the video screen so that the speaker is closer to the moderator. As an element (C), it is arranged on the video screen so that the first speech of each talker is closer to the moderator in an earlier order. Elements (A) to (C) described above
Using the above, the speaker placement analysis unit 3H extracts data for setting the speaker placement by the following predetermined calculation method.

Here, for the sake of explanation, the utterance of the speaker A is Ui (A) (subscript i is the order in which the text data is arranged in chronological order), and the utterance of B is Ui (B). It is written as. At this time, the text data is "A statement"
If the order is "B's remarks" and "A's remarks", then [U1
(A), U2 (B), U3 (A)].

When expressed using these numerical values, the element (A) is mini Σ _{(i = 0} to _k) 2sin (hπ / 8) (where k = the number of speakers and h = [Ui-1 ( A), Ui (B)] = distance between speaker A and speaker B when the diagonal line of the octagon in virtual space is set to 1, that is, for all possible arrangement combinations Select the sitting pattern that minimizes the total sum of the distances between performers calculated along the time series data,
Decide on this. When there are two or more layout patterns with the same total sum, the determination method in element (B) is applied.

In order to determine the arrangement by the element (B), for all the elements of the text data arranged in time series,
The number of appearances of the element is counted for each speaker, and the element having a large number of appearances is placed near the seating reference point. The determination method in element (C) is applied to target speakers with the same number of element appearance counts for each participant.

In order to determine the arrangement by the element (C), the elements of the text data arranged in chronological order are sequentially examined, and the speaker in which the element appears first in the utterance of the target speaker is arranged in order. Place it near the seating reference point.

As described above, the speaker placement analysis means 3H selects and positions the performers based on the method described above.
By determining the spatial arrangement, it becomes possible to express the positional relationship of the speakers with a reasonable camera angle in the image generation / display means 4 to be described later, and the speakers are connected to each other at the time of visualization. It is only necessary to set the conversation axis, which is an axis, to the minimum, and it is possible to suppress unnecessary switching (shot switching) beyond the conversation axis to the necessary minimum.

The speaker analysis means 3a, the speaker name quote analysis means 3d, the statement content quote analysis means 3e, the facial expression data analysis and extraction means 3f, and the gesture analysis means 3J perform morpheme analysis and character strings. Each data is extracted by matching or character string template matching.

Also, here, the TVM is used by using the result of the text data obtained by analyzing the conversation data by each means.
By using the L conversion means (not shown), it becomes a video corresponding to the CG character prepared in advance,
We are creating a TVML script.

For example, the speaker with the identification number A adds "Good" to the text data indicating the content of the comment as conversation data.
When the data "evening" is received, it is converted to a character string and "character: talk (name = A, text =" G
fixed () and pass it to an application called TVML player,
Finally, the screen as shown in FIG. 3 is generated so that the image in which the CG character speaks "Good evening" and the synthetic voice are output at the same time. Although illustration and detailed description of this synthesized voice are omitted, conversation data (text data and text data can be generated by a general voice synthesis device (voice synthesis means) included in the speaker specific video device 1. Synthetic voice is generated from the voice data) and added so as to be synchronized with the motion of the CG character.

Further, the gesture analysis means 3J uses each CG.
Automatic addition of character gestures, addition of gestures that encourage continuation of speech, and gesture reproduction of emotional expressions When each CG character becomes an image in the TVML format, it is reflected as an image with an expression or expression on the CG character. It is for analyzing and extracting data from conversation data.

<Automatic Addition of Gesture> Here, based on the result analyzed by the gesture analyzing means 3J,
An image generated by the image generation / display unit 4 will be described.
In the image generation / display unit 4, a gesture is automatically added to the generated image according to the analysis result by the gesture analysis unit 3J (automatic addition of gesture). For example,
The automatic addition of gestures described here is for performing an effect of adding a discussion program by CG characters to make it look more natural, and the analysis data (analysis result) required from the conversation data by the gesture analysis means 3J.
Has been extracted.

<Addition of Gesture for Encouraging Continuation of Statement> The discussion program is basically conducted in the form of face-to-face conversation. Therefore,
A video including a CG character automatically generated by the video generation / display unit 4 also automatically generates a program in a format in which a performer faces and discusses.

Generally, in a face-to-face conversation, when the utterance is long, the talker emits a short clause called “ac”. Specifically, the "ac" is an action indicated by a hammer, a nod, a smile, or the like, which accepts the speech of the other party and encourages continuation of the speech. By giving an "ac", the talker indicates to the speaker that he / she is listening to the talker and recognizes that the talker is still in the middle of the talk. Has a role to show that. It is reported that 77% of such "acs" are emitted near the end of a story or a grammatical break, and 45% of them are covered by the talk of the speaker. Also, in long conversations of 30 words or more, "ACK" occurs within a major interval of about 9 words, and about 80% occurs at least once in 15 words.

On the basis of these statistical data, the gesture analysis means 3J, when analyzing the conversation data, TVML
When a TVML script is generated by a conversion means (not shown) (in a kanji-kana mixed sentence), a long talk of 50 characters or more nods the dialogue participant and the face-to-face conversation is conducted naturally. The analysis data (analysis result) is extracted as described above. In the gesture analysis means 3J, the nod timing is calculated by detecting the end of the story or the grammatical break from the conversation data for each text of about 50 characters, and giving the variation with a random number to the nod motion of the CG character. Analysis data is extracted so as to be added.

<Reproduction of Emotional Gesture> In recent years, a unique emotional expression format has been widely used in Internet media such as electronic mail, chat, and bulletin boards. Typical ones are shown below. "Smiley":
A combination of [:-P] symbols to express facial expressions, and [! +] And [? +] (+ Is an arbitrary number of 0 or more)
Example of emphasizing emotional expression by stacking multiple symbols) "Why ???
? , Or [[vowel + (at the end of sentence)] or [sentence] [ー
+] Emphasis of emotional expression by extending the ending Example) "Uhyaaaaa!", And (laughs) (crying), etc.

It is said that these emotional expressions have a role of expressing the emotions of the speaker, softening tight expressions, expressing fine nuances, and expressing emphasis. Therefore, when these emotion expression formats (hereinafter referred to as expression data) are included in the text data, the gesture analysis unit 3J analyzes and extracts the expression data. Based on the analysis result, the image generation / display unit 4 can add an appropriate preset gesture to the CG character for each expression form. Note that the facial expression data of the CG character can also be enriched by analyzing and extracting facial expression data that changes the facial expression of the CG character.

The facial expression or gesture of the CG character may be added by transmitting and receiving the data tagged with the conversation data input in advance by the input unit 8 of the terminal. As a method of adding this tag, a predetermined mark is added to the utterance content of the text data, and a mark expressing emotional expression (in this example, it is enclosed by a tag) is added at the time of utterance, and it is added. Send and receive. When this method is adopted, the data analysis and extraction section 3 is provided with a step of detecting this mark and a step of reflecting what is detected by the data image generation and display section on the CG character.

When the conversation data is converted into metadata in XML format and transmitted / received, the data analysis / extraction section 3 detects tags using an existing XML parser (not shown), and this tag information is also imaged. It may be passed to the generation / display unit 4, and the image generation / display unit 4 may perform character facial expression generation which is provided as a function of TVML. Furthermore, a method of detecting and adding an emotional expression from the utterance content by character string matching may be used.

Next, the image generation / display section 4 will be described. The video generation / display unit 4 displays the video including the CG character that is the speaker agent of the speaker in the virtual space based on each data received from the data analysis / extraction unit 3 in the video display unit of the terminal owned by the speaker. It is for displaying on 7. The video generation / display unit 4 has a screen configurator 4a for determining a screen configuraton of a video including a CG character that is a speaker agent of a speaker, and a transition probability and a speaker of the screen configurator 4a. Screen transition means 4 for transitioning the screen configuration based on the statement length data
b and a screen transition timing means 4c for determining the timing of this screen transition. The video generation / display unit 4 is configured to analyze the data relating to the actual dialogue program in the debate programs photographed in the past and reflect the result in the video created as the dialogue program data.

The interactive program data is stored in the storage unit 5. In addition, the CG character is previously CG
Is used, or the one prepared by the speaker is used to reflect the image on the TVML.

In this embodiment, as the interactive program data, a video is created from a third-party viewpoint so that the discussion can be viewed objectively from the viewer's line of sight when the discussion program is produced. From the information shown, the screen layout was further improved to make it easier to understand the positional relationship between the speakers, and the insertion of shots suggested the relationship between the speakers and the effect of attracting the viewer's interest. Are carefully selected and stored in the storage unit 5. In addition, these productions are usually one of the important factors in making a video, and are performed based on the experience of the program producer.
This empirical rule is configured so that it can be reflected in the video generation / display unit 4 as interactive program data.

More specifically, in order to statistically calculate this empirical rule as the interactive program data, the actually broadcasted discussion program 42 discussions, 30 hours, 9000 cuts are analyzed from various angles, and Using the analysis results, the screen composition is obtained by statistically calculating the rules of "screen composition", "screen transition (shot determination)", and "screen transition timing (switching timing)" as the main effects related to visualization. The means 4a, the screen transition means 4b, and the screen transition timing means 4c are configured so that they can be reflected in the images generated.

When statistically calculating the rule of thumb of the program producer, in order to avoid subjective effects and addition of effects due to incorrect text interpretation, effective effects are achieved using only the surface features of the text. The result of the analysis is reflected as the calculation method for giving.

Here, the interactive program data will be described. The dialogue program data is constructed by connecting one shot for each statement in order along the flow of the dialogue using a screen configuration that takes one shot (1S) of the speaker for each statement of the dialogue by a plurality of speakers. It refers to the basics of camera work in discussion programs that compose the program. In this one-shot, the angle from the angle where the line of sight is removed is used so that the performer's remark can be seen from an objective standpoint.

In addition to the one-shot, in order to direct the viewer so that the flow of the dialogue can be understood smoothly,
It is necessary to take multiple shots in which the positional relationships of the performers are easy to understand. These multiple shots include a shot taken from the same side of the conversation axis that connects the speakers with a straight line (same conversation axis), or a shot on the screen with a space in the direction of the speaker's line of sight (speaker line-of-sight direction). Shot) etc. Returning to the description of the image generation / display unit 4.

The screen constructing means 4a is constructed to automatically construct a screen for each CG character based on the data relating to the screen configuration extracted by the data analysis / extraction section 3 along with the interactive program data. Has been done.

Specifically, in the screen composing means 4a, for example, when the composed screen is "one shot of the speaker", the vertical angle of view of the camera for photographing this speaker is θ = 2ta.
n-1 (l / 2rD) (l is the vertical direction of the screen, D is the distance r between the camera and the subject, normalized to the screen height of 1 (r = 0.6 for one shot)) Video with various screen configurations is generated.

The processing performed by the screen structuring means 4a includes, for example, speaker name quotation data, which is a quote of a participant's name, and comment content, which is a quote of the comment content of another speaker. If there is quoted data, as an example, there is line-of-sight processing that makes the screen in the direction in which the relevant speaker is present open (shifts by 20% from the center), and if not, the screen in the direction in which the previous speaker is present. There is a process for generating a screen that reminds me of.

Further, in the case where the screen to be constructed by the screen composing means 4a is other than "one-shot of the speaker", for example, when a shot around the speaker, which is a picture of the speaker and the speaker, is taken as an image, that is, In this "Speaker's Shot",
If the utterance content includes speaker name citation data that is a citation of the name of the talker, and utterance content citation data that is a citation of the utterance content made by another speaker, includes the relevant talker and speaker. A video having a screen configuration in which the minimum angle of view that can be captured by the minimum number of people and the minimum angle of view that can be captured by the minimum number of people including the former speaker is generated.

The screen transition means 4b transitions the screen constructed by the screen construction means 4a based on the transition probability and the speech length data of the speaker. For example, when the screen constructed by the screen composing means 4a is "speaker's one-shot", and this "speaker's one-shot" is connected to form a debate program, when one statement is long, etc. , The viewer gets bored with only a fixed shot for a long time. Therefore, the screen transition means 4b appropriately moves other shots for the purpose of giving movement and change to the screen in order to keep the viewer's interest. The screen configuration is transitioned to (the other shots are inserted).

In this screen transition means 4b, when a shot is inserted, unnecessary abuse of the shot may cause viewer's confusion, and short cut switching causes tension to the viewer. The transition of the screen configuration is set in consideration of the fact that the continuous camera movement using panning or the like makes the screen annoying and is not suitable for discussion programs.

In other words, the screen transition means 4b appropriately inserts a long shot to show the surroundings, a shot capturing a participant who reacts to the utterance, and the like, and establishes the mutual relations between the appearing speakers. Obviously, it is necessary to select shots that will change the screen while attracting the attention of the viewer.

Therefore, when the types of shots in the actual discussion program are roughly classified by paying attention to the performers in the video,
(1) One shot (1S) of the speaker (speaker),
(2) Shot of a plurality of people including a speaker (shot around the talker), (3) Speaker 1S, (4) Shot of a plurality of people including a talker (shot around the talker), (5) Since it is divided into dolly shots, the screen transition means 4b is configured to transition the screen configuration of each of the shots (1) to (5) in consideration of these things.

In the screen transition means 4b, the screen construction means 4a
For each of the shots (1) to (5) that is the screen configuration data screen-configured by
The transition probability of (5) is set. The transition probabilities are 55%, 16%, 11% for each shot (1) to (5) as a result of the statistical processing based on the interactive program data.
The ratio was 7% and 10% (the number of cuts in each shot / the total number of cuts). Note that shots (1) to (5) are 70%, 19%, 1%, 5%,
It was found that the transition probability that the speaker was copied was high at 5%, and that the types of shots inserted after that were particularly high in relation to the immediately preceding shot. These relationships are shown as Table 1 and reflected in the flowchart of FIG. In Table 1, the numerical values in the first column are reflected by A1 to A5 as shown by A1 to A5 surrounded by a dotted line in FIG. Similarly, in the second and subsequent columns, A1 surrounded by a dotted line in FIG.
Reflected in the numbers below A5.

[0074]

[Table 1]

The screen transition timing means 4c determines the timing of transition of the screen to the next shot set by the screen transition means 4b. A method of calculating the shot switching timing performed by the screen transition timing means 4c will be described. If the timing of switching shots is determined based on only the types shown in the shots (1) to (5), the switching timing is determined in the case of the one-shot of the speaker, as a result of statistical processing based on the interactive program data. After investigating with real data,
What was within the predicted value of 16 seconds ± 12 seconds using the standard deviation was only 20%. This was the same for all shots. Therefore, we identified the main factors that determine the timing for switching shots by discussing the actual discussion program 42 discussion, 30 hours, 9
It started from 000 cuts.

As a result, it is assumed that the screen transition timing means 4c causes the screen transition to be appropriate at the time points described below. (1Y) When a superimpose or explanation flip is displayed due to the program structure (Su
(per), (2Y) when the name of the talker is quoted during the remark (name quote), (3Y) when the talker makes a gesture (gesture follow shot (gesture)), (4Y) one For example, when it is desirable for the production that the utterance is long and the screen is changed (long utterance), (5Y) the utterance of another participant is quoted (utterance citation).

Table 2 shows the results of an examination of the factors (1Y) to (5Y) and the duration of the shot with respect to the interactive program data (discussion program).

[0078]

[Table 2]

As shown in Table 2, for example, the speaker 1S
In the case of, the average duration of the shots subjected to the superimposition (Super) was 25.51 seconds, and the standard deviation thereof was 10.34 seconds. The duration of the shot (transition timing) was calculated using Table 2 as a reference. That is, the shot in which the superimpose (Super) is performed is randomly determined from the duration of 25.51 ± 10.34 seconds. Thus, (1Y) to (5
The factors up to Y) and the shot duration (objective variable) were calculated and reflected in the screen transition timing. As a result, the prediction accuracy could be increased up to the multiple correlation coefficient of 0.83 (prediction accuracy of 70%), so these factors (1Y) to (5Y) were extracted from the following items.

That is, as a method of extracting the factors that improve the prediction accuracy, (1a) Insert a superimpose at the time of each speaker's first speech and refer to the reference URL in the input text.
When there is a description flip insertion, (1b) extraction of a factor by pattern matching between the input text input and the speaker name, (1c) operation timing of a CG character gesture (described later) automatically assigned, (1
d) The length of the input text is standard speed (150 characters /
(1e) Extraction of factors by pattern matching between input texts. Note that the prediction accuracy indicates the degree of reproducibility similarity with respect to the configuration of the discussion program actually shot.

Then, the factors (1Y) to (5Y) are extracted by the methods (1a) to (1e), and the residuals corresponding to each factor (each The switching timing (screen transition timing) of the screen transition timing means 4c is determined by providing variation in the variance of shot prediction residuals).

As described above, in the video generation / display means 4, the discussion program 42 that was actually broadcast in the past, discussion, 30 hours, 9
From 000 cuts (interactive program data), the image configuration means 4a, the image transition means 4b and the image transition timing means 4c set the image configuration, the image transition, the image transition timing, etc., and the CG character data of the speaker is set in the TVML format. The included image is generated and displayed on the image display screen section 7 of the terminal.

The storage unit 5 is for storing each data, and is not particularly limited as long as it can store data such as a hard disk.

Next, with reference to FIGS. 1 to 6, with reference to FIGS. 2 and 3 and FIG.
The action of will be described. FIG. 6 is a speaker identification video device 1.
3 is a schematic diagram for schematically explaining the overall flow of information in FIG. As shown in FIG. 6, a schematic flow of the speaker identification video apparatus 1 will be described first. In the speaker identification video device 1, a speaker inputs conversation data (text data) from the input unit 8 of the terminal via the network 6 and the data transmission / reception unit 2 (see FIG. 1). Then, the data analysis / extraction unit 3 of the speaker identification video device 1 creates a TVML script from the text data. In the data analysis and extraction unit 3, the conversation structure analysis and extraction unit 3A extracts the structure data such as the utterance order, the number of times, and the length, and the conversation content analysis and extraction unit 3D and the gesture analysis unit 3J.
By this, content analysis data such as name citation and emotional expression is extracted.

Then, in the video generation / display unit 4 of the speaker-specific video apparatus 1, the structure data analyzed by the data analysis / extraction unit 3, the content analysis data, and the TVML script are used to produce an effect (shot / time [transition timing] / gesture). Is determined, and the video of the CG character is output controlled as an interactive program through the control module.

Next, the operation of the speaker specific video device 1 will be described in detail. First, each speaker inputs text data or voice data from the input unit 8 of the terminal as conversation data, and then, via the network 6,
The conversation data is transmitted to the data transmitting / receiving unit 2. When the conversation data is received, each unit of the data analysis / extraction unit 3 analyzes and extracts necessary data. In the data analysis and extraction unit 3, performers (speakers and speakers) are determined (selected) in the order of the input conversation data based on the extracted identification data and the serial number. In this embodiment, the first speaker is set as the first speaker, and the first speaker is set as the moderator.

That is, in many of the conventional dialogs such as those using the electronic bulletin boards and mailing lists, there are many cases in which a community is formed accidentally, and normally, clear roles are shared among the speakers. Absent. Therefore, in the speaker specifying video device 1, in the conversation data input from a plurality of speakers, the speaker who first speaks has a role of giving an opportunity to direct the entire dialogue unfolded by the plurality of conversation data. I decided to treat it as a moderator.

Further, in many of the accidental dialogues on the network, in many cases, an unspecified number of speakers are speaking, so this unspecified number of speakers are all included in one program (virtual If (appearing) as CG characters appearing (in space), the number of viewers becomes too large, and it becomes difficult for the viewer (speaker) to identify the performers (speaker, talker). There is a risk that

It is said that the maximum value at which a human can simultaneously identify a large number of objects having similar properties and shapes and can be identified and recognized without difficulty is 7. Therefore, in the data analysis and extraction unit 3,
We decided to limit the maximum number of speakers to 7 people. When seven or more speakers are present in the text data, the seven speakers with the highest utterance frequency are determined as performers. Regarding the utterances of speakers other than the above, the above-mentioned moderator makes an utterance on behalf of the speaker to proceed with the program.

Next, the data analysis / extraction section 3 of the speaker-specific video device 1 determines the spatial arrangement positions of the performers (speakers, speakers) based on the extracted identification data and the serial numbers. . When determining the spatial arrangement of performers (speakers, speakers), the maximum number of speakers is 7 as described above, so add a camera that captures the main video and arrange them on an octagon. Things are the basis. In this case, as shown in FIG. 5, the moderator is placed at a position facing the camera. By arranging in this way, it is possible to sit at equal intervals when the maximum number of people is present, and it is possible to prevent the relative positions from becoming difficult to understand because the intervals between speakers who appear in a small number of people are too wide. Further, when the number of speakers is small, by placing the speakers at a position adjacent to the moderator, many speakers can be taken even with a narrow angle of view (see FIG. 5).

Further, the data analysis / extraction section 3 determines the spatial arrangement order of speakers who appear. Using the position of the moderator (first speaker) as the standard placement, examine the utterance data arranged in chronological order, and based on the predetermined elements (A), (B), and (C) already explained. The placement order of speakers is determined.

That is, as shown in FIG. 6, necessary data is extracted from the conversation data by analyzing the conversation data so that the spatial arrangement order of the speakers can be determined, and the CG character displays the interactive program data via the video generation / display unit 4. Is displayed (output) in a state (interactive program) configured in accordance with. It should be noted that in this embodiment, T is calculated from the input conversation text data.
A system for automatically generating a V discussion program was realized on a personal computer as a speaker identification video device 1.
As shown in FIG. 6, the spatial arrangement of speakers and the arrangement order are determined from the input conversation data (text data), and based on this, the initial setting of the CG character T is displayed.
A VML script is generated.

At the same time, a TVML script showing a scene in which the CG character in the program speaks conversation contents is generated. By simply connecting the above two TVML scripts into one script and playing it back on the TVML player,
A scene is created in which the appropriately arranged CG characters discuss.

In addition to the above, the screen forming means 4a,
Screen transition means 4b and screen transition timing means 4c
Thus, camera switching and CG character gesture are performed. In this embodiment, these are not described directly in the TVML script, but are realized by using the external interrupt function of the TVML player to give camera switching and gesture triggers in real time.

FIG. 6 shows a part of the TVML script. T
The VML player reads this script line by line and (50
Line): The CG character Rabi is "Good e
The scene that says "vening" is generated in real time,
Output with synthetic speech. (Line 51): Rab continues
Generate a scene in which the camera closes up at i.
(Line 52): Rabi creates a bowing scene.
(53rd line): Frame 0 of moving image file called tmp.mov
The program is played back from 1 to 200. As described above, TVML is a language that can describe not only CG characters but also all elements necessary for a television program, such as moving images, still images, superimposes, and voices. In the speaker specific video device 1, the output video / audio is generated by using the TVML language.

Based on the result of analyzing the text data input from the camera switching generator (not shown) and the gesture generator (not shown), these effects are generated and a trigger is transmitted to the TVML player. Achieve the desired effect. In addition, TVML
The player's external interrupt function is a TVM
While playing the L script, a line of TVML script from an external application (eg camera close-up)
Is transmitted to the TVML player via the shared memory and the interrupt operation is performed.

Then, in the speaker identifying video device 1, the speaker-identifying video device 1 shown in FIG.
As shown in, a moving program (video) with camera switching and gestures is generated as a video by TVML in accordance with the screen configuration, the screen transition and the screen transition timing, and the configuration of the interactive program data.

Here, as the transition probabilities of the interactive program data, the ratios of 55%, 16%, 11%, 7% and 10% (cut number of each shot / total cut number) are used. Furthermore, if it is limited to the start of remarks, it is also reflected that the shots (1) to (5) have a high transition probability of 70%, 19%, 1%, 5%, and 5%, respectively. I did it. The types of shots to be inserted after that are also reflected in the fact that the relationship with the immediately preceding shot is particularly high.

As the timing of the screen transition timing means 4c, one shot is visualized and the standard deviation of 10.84 is added to the average of 69 seconds by a random number. When the determined shot is in the vicinity of the speaker, the utterance content character count is s
Then, a standard deviation of 0.048 s is added to the average of 0.08 s as a random number. When the shot determined in the screen transition other than the first utterance is the one shot of the speaker, the one shot is visualized and the standard deviation is 8.36 at an average of 13.25 seconds.
Is added by a random number.

Similarly, for the speaker-containing peripheral shots, the standard deviation 4.51 is added to the average of 7.125 seconds by random numbers. Also, for the dolly shot, the standard deviation 8.911 is randomly added to the average of 15.54 seconds. Furthermore, for the shots around the talker, a standard deviation of 3.206 is randomly added to 4.96 seconds on average. Then, the talker one-shot randomly adds the standard deviation 2.386 to 4.56 seconds on average. In this way, the screen transition timing of the screen transition timing means 4c is set.

As shown in FIG. 2, for example, at the start of a speech, a random number of the speaker 1S, a plurality of shots including the speakers (around the speaker), a speaker 1S, and a speaker 1S are selected. 0.70 / 0.19 / 0.01 / including multiple shots (shots around the talker) and dolly shots
Images taken at a ratio of 0.05 / 0.05 are used.
For the subsequent insertion shot, the video decided by the previous shot is used.

After the speaker's 1S, the speaker's 1S, the speaker's peripheral shot, the participant's 1S, the participant's peripheral shot, and the dolly shot are 0.11 / 0.32 / 0.35 / 0.15. /
0.08, and after the shots around the speaker, 1S of the speaker, shots around the speaker, 1S of the talker, shots around the talker, and dolly shots are 0.85 / 0.04 / 0.08.
/0.02/0.1.

After the participant's one shot, the speaker 1S, the speaker peripheral shot, the speaker 1S, the speaker peripheral shot, and the dolly shot are 0.72 / 0.06 / 0.
19 / 0.02 / 0.01, and after the talker's peripheral shot, the talker's 1S, the talker's peripheral shot, the talker's 1S, the talker's peripheral shot, and the dolly shot are 0.73 / 0. 1
4 / 0.08 / 0.02 / 0.01, after the dolly shot, 1S of the speaker, a shot around the speaker, 1S of the talker,
Shot around the talker and dolly shot 0.73 / 0.
Insert at a ratio of 11 / 0.11 / 0.04 / 0.01.

[0104] When the one-shot of the talker is decided,
If there is a name citation or utterance quoter in the content analysis data, that speaker is visualized. If not, the previous speaker is visualized. Similarly, when a shot around the talker is decided, the name is quoted in the content analysis data, the speaker is included if there is a quoter, and if not, the speaker is included, including the previous speaker. Calculate and visualize the shots that do not exist.

In the screen transition timing means 4a,
Specifically, the screen transition timing is based on the following timing. That is, when each speaker speaks for the first time and the shot determined in the screen transition is one shot of the speaker, the one shot is visualized and the standard deviation 10.84 is added to the average 69 seconds by a random number. When the determined shot is in the vicinity of the speaker, when the number of utterance content characters is s during the utterance time, the average is 0.08s and the standard deviation is 0.048
Add s with a random number. If the shot determined in the screen transition other than the first utterance is the speaker's one shot,
Visualize one shot and add a standard deviation of 8.36 to the average of 13.25 seconds by random numbers.

Similarly, for the speaker-containing peripheral shots, a standard deviation of 4.51 is added to the average of 7.125 seconds by a random number. Also, for the dolly shot, the standard deviation 8.911 is randomly added to the average of 15.54 seconds. Furthermore, for the shots around the talker, a standard deviation of 3.206 is randomly added to 4.96 seconds on average. Then, the talker one-shot randomly adds the standard deviation 2.386 to 4.56 seconds on average. In this way, the screen transition timing of the screen transition timing means 4c is set.

The screen configuration, screen transition, screen transition timing, utterance content, identification number, C determined by the above-mentioned configuration
Using the G character, an image in which the CG character speaks is generated in the order of the speech serial numbers.

As shown in FIG. 3, description will be made based on an example of an image actually generated. First, as the conversation data handled by the speaker identification video apparatus 1, the title 100 is displayed as an interactive program format in which the electronic bulletin board is represented by CG characters.

Here, the identification number A (the first speaker is (R
abi) Displayed as a rabbit CG character) is the moderator, and when the conversation data "Good evening" is received, it is converted into a character string and converted to "char.
acter: talk (name = A, text = ”Good evenin
g ″) and pass this to an application called TVML player to generate the screen of FIG. 3, and simultaneously output a video and a synthetic voice in which the CG character speaks “Good evening”. A screen configuration and a screen transition timing, which is a screen switching timing, are added to this video to add an effect for promoting the identification of the speaker.

[0110] Next, from the dialogue such as this time's dialogue program and the mailing list, five speakers can talk in this time. The first is to show a shot around the speaker in which the entire image is displayed by the screen composing means 4a. A video screen 101 is shown.

On the first video screen 101, the rabbit CG, which is the speaker agent of the speaker A who made the main statement of this conversation, is diagonally across the camera position at the center of the octagon. A rabbit character Uc, which is a character, is arranged as a moderator. Here, the video is configured so that the conversation content is displayed by characters in the lower frame of the video.

Then, as shown on the second video screen 102, first, a greeting or the like by the rabbit character Uc is displayed in one shot, and the character data is displayed together with the video of the rabbit character Uc. At this time, the rabbit character Uc is displayed for a predetermined time as if the rabbit character Uc were talking about the character data by the expression corresponding to the character data. As shown in FIG. 4, not only for the rabbit character Uc but also for other CG characters, it is possible to change the gesture according to the input expression data or emotion expression data, or to change the facial expression normally,
It can be converted to show emotions.

Further, as shown in the third video screen 103, when the rabbit character Uc speaks to the girl character Lacta Gc who is the speaker agent of the speaker B who is seated on the left side, the rabbit video character Uc is also displayed on the first video screen 101. As can be seen, in the positioning considering the overall configuration, the story continues while facing the girl character Lacta Gc side. Regarding the screen configuration, the screen transition, and the screen transition timing from the second video screen 102 to the third video screen 103, as shown in FIG. 2, they are switched and displayed in accordance with the configuration of the interactive program data prepared in advance. There is.

That is, as shown in FIG. 2, the shot of S1 = speaker 1S is selected through a random number, and the shot of S2 causes the screen configuration to be displayed on the second video screen 102. Then, the identification symbol is judged as S4 through the route of S3. Since the identification number is already known this time, the second video screen 102 is displayed for a time of “waiting for switching timing = 69.0 + standard deviation value (10.84)” from the Yes path, and then shot through a random number. Is determined. This time, as shown in S5, the third video screen 103 is selected as shot = speaker vicinity, and S6 is selected.
In the branch of, it is judged that "there is speaking time remaining". In the present example, the route in which “there is remaining speech time” is Yes, and the third video screen 103 is displayed again as “shot = periphery of speaker” selected in S5 by “shot” in S2. .

Similarly, regarding the screen configuration, the screen transition, and the screen transition timing, the fourth video screen 104, the fifth video screen 105, and the respective video screens are determined according to the flowchart of FIG. 2, which is the configuration of the interactive program data. , Displayed with character data. In addition, since it is understood that the speaking time is short on the fifth video screen 105, in FIG.
The next video screen is determined by the route of. In FIG. 3, a shot for the girl character Gc = speaker 1S is displayed as the sixth video screen. It should be noted that the girl character Gc shown on the sixth video screen also has a composition in which the arrangement of all the participants is always taken into consideration, and the girl character Gc is displayed so as to give its own opinion to the rabbit character Uc. In addition, each CG character can also nod or change his / her facial expression during the conversation.

As described above, performers (speakers, speakers) are composed of the screen configuration (shot) data, the screen transition data, the screen transition timing data, and the interactive program data.
In, it becomes possible to have a conversation in a state in which it is easy to objectively recognize who is talking to whom.

Although the speaker identification video device has been described here, the operation of each unit is described as a computer program language, and the main control unit (CPU etc.) of the computer is described.
It is also possible to develop it into a program and use it as a program that functions by utilizing each data stored in the storage unit 5.

[0118]

Since the present invention has the configuration relating to the speaker specifying video device, the program thereof, and the speaker specifying video display method as described above, the following excellent effects are obtained. . According to the inventions of claim 1, claim 6, and claim 7, the speaker of the conversation data is CG from the conversation data which is text data or voice data.
Visualization as a character makes it easier to identify the speaker and more easily understand the content of the conversation, as compared to the conventional transmission and reception of conversation data of only text data or audio files. It becomes possible to make a video with a sense of reality. Further, since no information other than the identification ID of the speaker and the text (or audio file) describing the content of the statement is used, it can be applied to various applications.

Further, according to the inventions of claim 1, claim 6, and claim 7, in asynchronous communication such as Internet chat and electronic conference, even if it is difficult to intuitively understand the speaker, The speaker can be easily understood by the image of the CG character in consideration of the arrangement, screen configuration, and the like. Further, even when a video image of a video conference is used, it is possible to automatically adjust the timing and size of switching the camera by referring to the interactive program data, which has a wide range of applications.

Further, when actually implemented as a program, the conversation data in the mailing list is input and converted into a discussion program consisting of video and audio, so that it is possible to actively read, click, etc. It is possible to convert enjoyable text data contents into passively enjoyable TV contents such as "watch" and "listen".

In general, since it becomes possible to convert the conversation data by a plurality of speakers into a television program,
Various applications are possible. For example, contents developed in various media such as conversation texts on WEB pages, electronic bulletin boards, mailing lists, chats, dialogue texts in magazine manuscripts can be viewed on a television, and the conversation content can be in the form of a CG character's statement. Therefore, it is possible to make the image more recognizable. Also, a discussion program by CG characters can be generated by using the analysis result of the video composition in the discussion program, knowledge about the spatial arrangement of the performers, gestures, etc. Similarly, by a still image / video avatar in the virtual space. It can also be applied as a direction addition method to remote conferences and automatic dialogue program recording by robot cameras.

According to the second aspect of the present invention, the conversation structure analysis and extraction means analyzes and extracts the statement length data and the serial number data from the conversation data as the structure data, and the conversation content as the content analysis data. Analysis and extraction means
The speaker name citation data and the utterance content citation data are analyzed and extracted from the conversation data.
You can give the character a natural motion.

According to the third aspect of the present invention, the CG character of the speaker generated in accordance with the screen structuring means, the screen transition means and the screen transition timing means provided in the video generation / display section, and the configuration of the interactive program data. Thus, it is possible to provide an image in which it is easy to understand which speaker is talking to whom and which is easy to recognize.

According to the invention of claim 4, since the CG character of the speaker can be given a richer expression, the viewer who is viewing the video can provide it as a more natural dialogue video.

According to the fifth aspect of the invention, the speaker placement analysis means determines the placement of the CG character based on the content analysis data, and the CG character playing a central role in the conversation is faced to the camera line-of-sight position. It is possible to determine the seating reference position to be performed and sequentially arrange the CG characters, such as the number of times of utterances, at positions close to the seating reference position. Therefore, by selecting the speaker, determining the position, and arranging the space, it becomes possible to express the positional relationship of the speakers with a reasonable camera angle in the image generation / display means, and at the time of visualization, It is only necessary to set the conversation axis, which is the axis connecting the speakers, to a minimum, and it is possible to minimize unnecessary switching (shot switching) beyond the conversation axis.

[Brief description of drawings]

FIG. 1 is a block diagram schematically showing a speaker identifying video device according to the present invention.

FIG. 2 is a flowchart for setting an image used in the speaker identifying image device according to the present invention.

FIG. 3 is a schematic diagram showing an image configuration and a screen transition state by the speaker identifying video device according to the present invention.

FIG. 4 is a schematic diagram schematically showing facial expressions of a CG character according to the present invention.

FIG. 5 is a schematic plan view showing a state in which the placement of speakers is set by the speaker identifying video device according to the present invention.

FIG. 6 is a schematic diagram schematically illustrating the overall flow of the speaker identifying video device according to the present invention.

[Explanation of symbols]

1 Speaker identification video device 2 Data transmission / reception section (data input section) 3 Data analysis and extraction section 3A Conversation structure analysis and extraction means 3a Speaker analysis means 3b Serial number analysis means 3c Speech length analysis means 3D conversation content analysis and extraction means 3d Speaker name quoting analysis means 3e Statement content citation analysis means 3f Expression data analysis and extraction means 3H Speaker placement analysis means 3J gesture analysis means 4 Video generation display section 4a Screen configuration means 4b Screen transition means 4c Screen transition timing means 5 memory 6 network 7 Video display section (display screen) 8 Input section

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Ichiro Yamada             1-10-11 Kinuta, Setagaya-ku, Tokyo, Japan             Broadcasting Association Broadcast Technology Institute (72) Inventor Masahiro Shibata             1-10-11 Kinuta, Setagaya-ku, Tokyo, Japan             Broadcasting Association Broadcast Technology Institute (72) Inventor Nobuyuki Yagi             1-10-11 Kinuta, Setagaya-ku, Tokyo, Japan             Broadcasting Association Broadcast Technology Institute F-term (reference) 5B050 AA08 BA08 BA12 CA08 EA19                       EA24 FA10                 5C064 AA02 AB03 AB04 AC04 AC06                       AC09 AC12 AC16 AD08 AD14

Claims (7)

[Claims] 1. When a plurality of speakers talk synchronously or asynchronously using a terminal connected to a network by text data or voice data to which identification data for identifying speakers is added, A speaker identifying video device for easily identifying a speaker, comprising: a data input unit for inputting conversation data of the text data or the voice data for conversation; and a conversation input to the data input unit. A data analysis and extraction unit for analyzing the data to extract the structural data showing the structural features of the identification data and the conversation data thereof, and the content analysis data showing the characteristic features of the conversation data; Based on the structure data and the content analysis data and the identification data that are analyzed and extracted by the analysis and extraction unit, An image generation / display unit that generates an image including the CG character of the speaker and causes the image including the CG character to be displayed on the display screen of the terminal according to the structure of the interactive program data prepared in advance. A speaker identification video device, comprising: a storage unit for storing the respective data. 2. The data analysis and extraction unit analyzes, as the structure data, at least statement length data indicating a statement length of a speaker and serial number data indicating a serial number of a statement order of the speaker. With the conversation structure analysis and extraction means for extracting, as the content analysis data, at least,
Conversation content analysis and extraction means for analyzing and extracting speaker name citation data indicating whether or not a speaker name is quoted, and utterance content citation data indicating whether or not the utterance content of another speaker is cited. The speaker identification video apparatus according to claim 1, wherein the speaker identification video apparatus is provided. 3. The image generation / display unit is based on a screen structuring unit that determines a screen configuration of an image including the CG character, a transition probability of the screen configuration of the screen structuring unit, and the speech length data of the speaker. 3. The speaker specifying video device according to claim 1, further comprising: a screen transition means for transitioning the screen configuration and a screen transition timing means for determining a transition timing of the screen transition means. 4. The data analysis / extraction section includes facial expression data analysis / extraction means for extracting facial expression data for changing the facial expression of the CG character based on the conversation data. Item 10. The speaker identifying video device according to any one of items 3. 5. The data analysis / extraction section arranges the CG characters at octagonal vertex positions including the camera line-of-sight position based on the conversation data, and
Based on the content analysis data, a seating reference point of the CG character is determined at a position facing the camera line-of-sight position, and a speaker placement for sequentially determining the placement of the remaining CG characters to the left and right of the seating reference point. The speaker-specific video device according to claim 1, further comprising an analyzing unit. 6. When a plurality of speakers perform a synchronous or asynchronous conversation using a terminal connected to a network by text data or voice data to which identification data for identifying the speaker is added, Data input means for inputting conversation data based on the text data or the voice data for conducting a conversation, in order to generate a speaker identification image for easily identifying the speaker, and input to the data input means Data analysis and extraction means for analyzing the identified conversation data and extracting structural data indicating the structural characteristics of the identification data and the conversation data, and extracting content analysis data indicating the characteristic characteristics of the conversation data. The structure data and the content analysis data and the knowledge extracted by being analyzed by the data analysis and extraction unit. An image including the CG character of the speaker is generated based on the data, and the image including the CG character is displayed on the display screen of the terminal according to the structure of the interactive program data prepared in advance. A speaker identification video program characterized by causing it to function as video generation and display means. 7. When a plurality of speakers talk synchronously or asynchronously using a terminal connected to a network by text data or voice data to which identification data for identifying speakers is added, A speaker identification video display method for easily identifying the speaker, comprising inputting conversation data according to the text data or the voice data for conversation, and analyzing the conversation data to identify the identification data. And extracting structural data indicating the structural characteristics of the conversation data, and extracting content analysis data indicating the characteristic characteristics of the conversation data, the structural data, the content analysis data, and the identification data Based on the generated CG character of the speaker, and including the CG character An image along the configuration interaction program data are prepared in advance, speaker specific image display method which comprises the steps of: displaying on the display screen of the terminal.