JPH05313686A - Display controller - Google Patents

Abstract

PURPOSE:To output an optional voice and an image synchronously. CONSTITUTION:A voice synthesis part 12 calculates parameters for synthesizing a voice corresponding to a text supplied from a application program 14 through a control part 11. Then, when a mouse is operated and a speak button is clicked, a control part 11 outputs an actuation instruction to a voice synthesis part 12 and an animation display part 13, and the voice synthesis part 12 outputs the parameters for synthesizing the voice corresponding to the text to a voice synthesizer, which synthesizes the voice according to the parameters. At the same time, the animation display part 13 outputs an animation corresponding to the motion of the mouse for vocalizing the text in order.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a display controller suitable for application to, for example, a CAI system for learning English pronunciation.

[0002]

2. Description of the Related Art In a conventional CAI system, an audio signal and an image signal previously recorded in, for example, a hard disk are read out, and an image corresponding to the audio (audio corresponding to the image is synchronized with a predetermined synchronization signal). ) Can be output.

Thus, for example, the English uttered by the native speaker is recorded in advance in the hard disk as an audio signal, and the animation of the mouth movement (motion) when the English is uttered as the image signal is recorded in the hard disk in advance. If recorded, the user can learn the pronunciation of English and the movement of the mouth when the English is uttered at the same time.

[0004]

As described above, the conventional C
In the AI system, it is necessary to record the audio signal and the image signal in the hard disk in advance. Therefore,
For example, there is a problem that only a finite number of voice signals and image and voices and images corresponding to image signals recorded in units of words can be output, and the range for learning by the user is limited.

Therefore, there is a method of sequentially adding audio signals and image signals corresponding to English words to be recorded on the hard disk. However, in this method, every time an audio signal and an image signal which are not recorded in the hard disk are found, that is, when the user is learning, the learning is interrupted and the audio signal and the image signal are added. Since it had to be recorded, there was a problem that attracted learners' interest.

The present invention has been made in view of such a situation, and it is possible to output an arbitrary sound in synchronization with an image.

[0007]

A display controller according to a first aspect of the present invention includes a voice synthesizer 1 and a voice synthesizer 12 as a voice synthesizer for synthesizing and outputting voice from text.
A CRT 3 and an animation display unit 13 as display means for displaying a moving image such as an animation corresponding to a text, a voice synthesized by the voice synthesizer 1 and a voice synthesis unit 12, and a CRT 3 and an animation display unit 13 are displayed. It is characterized by comprising a control unit 11 as a synchronizing means for synchronizing the generated animation.

This display control device can cause the CRT 3 and the animation display section 13 to display an animation in which the mouth moves in accordance with the text.

[0009]

In the display control device according to the first aspect, the voice synthesized from the text and the animation that moves corresponding to the text are output in synchronization. Therefore, it is possible to output any voice in synchronization with the animation.

When the animation in which the mouth moves corresponding to the text can be displayed on the CRT 3 and the animation display unit 13, the learner's interest can be attracted by applying it to the CAI system, for example.
The learning effect can be improved.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a C to which the display control device of the present invention is applied.
It is a block diagram which shows the structure of one Example of AI system. The terminal 2 is connected to the voice synthesizer 1 via an RS232-C cable 7, and is composed of a CRT 3, for example, a workstation (EWS) 4 for UNIX, a keyboard 5, and a mouse 6.

The EWS 4 is a CPU that controls the entire apparatus,
It is composed of a ROM that stores a system program and the like, and a RAM (none of which is shown) that stores data necessary for the operation of the apparatus, and is displayed on, for example, a UNIX operation system as an operation system and a CRT 3 ( For example, an X window system is installed as a control program for controlling (opened) windows and the like.

The CRT 3 is controlled by the X window system installed in the EWS 4 and displays, for example, an animation or a text (character string) input by the keyboard 5 in response to a signal output from the EWS 4. The keyboard 5 and the mouse 6 are connected to the EWS 4 and are operated when creating text or issuing a command to the EWS 4.

The voice synthesizer 1 uses RS232 from EWS4.
-A speaker (not shown) that synthesizes voice from the characteristic parameters of the voice supplied via the C cable 7 and is built in
Output more.

The CAI system shown in FIG. 1 is for learning English, and is an environment-providing type and communication type (both sides initiative type) system in which a learner can think actively by trial and error. Yes, ○ Simulation function ○ Search function ○ Has a user-friendly interface, and by using multimedia, it is possible to provide learning that is not bound by the range and various learning methods.

Further, this CAI system is composed of five subsystems called "STUDY", "POST OFFICE", "LIBRARY", "PLAYROOM", and "LABORATORY" in order to create a pleasant school atmosphere.

The subsystem "STUDY" learns the grammatical basics of English based on textbooks, understands the contents, and also carries out problem exercises to establish knowledge.
It is a basic learning room.

Subsystem "POST OFFICE"
Simulates writing and sending letters in English to help you learn practical English expressions and letter formats.

The subsystem "LIBRARY" is a support tool which has a database of words used in the system (each subsystem) and which can be freely searched / referenced from other subsystems. This subsystem also provides pictures and sounds so that the learner can capture the image of the words without the help of Japanese and with the aid of visual and auditory senses.

The subsystem "PLAYROOM" is intended to prevent the learner's interest in studying English from being lost, and he / she can play English games without hesitation. You are able to learn.

Subsystem "LABORATORY"
Is an experiment for conversation in English. By combining images and sounds, we create a situation (virtual situation) as if we are actually talking to a person, and let the learner experience English conversation in it. You can do it.

Subsystem "LABORATORY"
Is composed of an application 14, and a control unit 11 as a user interface for creating a situation (virtual situation) as if one were actually talking to a person, a voice synthesis unit 12, and an animation display unit 13 (see FIG. 2).

The control unit 11 controls the voice synthesizing unit 12 and the animation display unit 13, and the voice synthesized by the voice synthesizing unit 12 (voice synthesizer 1) and the animation display unit 13 are controlled.
To synchronize the movement of the mouth of the animation displayed on the CRT 3. Further, the control unit 11 counts the number of characters of the text supplied from the application 14, the number of commas (,), or the number of periods (.) Included in the text, respectively, and displays the animation display time (CRT3). The time for moving the mouth of the animation) is calculated and output to the animation display unit 13.
The voice synthesizing unit 12 calculates a parameter for synthesizing voice from a text (character string) supplied from the application 14 via the control unit 11, and outputs the parameter to the voice synthesizer 1. Under the control of the control unit 11, the animation display unit 13 reads the cell frame data for moving the animation (animation mouth) stored in the cell frame data storage unit 13a (FIG. 6) and outputs it to the CRT 3. An animation (animation mouth) is stored in the cell frame data storage unit 13a.
The cell frame data (FIG. 8) for moving the cell are stored.

The application 14 is an application in the subsystem "LABORATORY", and comprises a dialogue processing section 21, a language processing section 22, a knowledge database searching section 23, and a knowledge database 24, as shown in FIG. The dialogue processing unit 21 stores (holds) scripts and dialogue history relating to various situations.
And manage the flow of dialogue based on these.

The language processing unit 22 is composed of an analysis unit and a generation unit (neither is shown). The analysis unit of the language processing unit 22 performs syntactic analysis and semantic analysis of the English sentence input by the keyboard 5, converts the sentence into an internal representation, and supplies the internal representation to the knowledge database search unit 23. The generation unit of the language processing unit 22 uses the answer returned from the knowledge database search unit 23,
An English sentence is generated based on an instruction (control) from the dialogue processing unit 21 and output to the control unit 11 (FIG. 2).

The knowledge database search unit 23 searches the database stored in the knowledge database 24 for a response to the internal expression supplied from the analysis unit of the language processing unit 22 and outputs it to the generation unit of the language processing unit 22. .. The knowledge database 24 stores reply patterns to questions in the form of internal expressions.

Next, the operation will be described. First,
In the CAI system shown in FIG. 1, the keyboard 5 or the mouse 6 is operated and the application 1 shown in FIG.
4 is activated, an animation 32 as a teacher and an animation 31 as a user of the CAI system, that is, the learner himself are displayed on the CRT 3 as shown in FIG. Further, the keyboard 5 is operated, and the text 33 (What is the fastest train in Jap as a question sentence is displayed.
an?) is input, the text 33 is displayed on the window of the CRT 3 controlled by the X window system, and the language processing unit 22 of the application 14 is displayed.
The text 33 is provided (FIG. 3).

In the analysis section of the language processing section 22, the text 33 input by the keyboard 5 is subjected to syntax analysis and semantic analysis, converted into an internal representation, and supplied to the knowledge database search section 23. In the knowledge database search unit 23, the reply to the internal expression supplied from the analysis unit of the language processing unit 22 is searched from the database stored in the knowledge database 24 and output to the generation unit of the language processing unit 22. In the generation unit of the language processing unit 22, the answer returned from the knowledge database search unit 23,
Text 3 based on an instruction (control) from the dialogue processor 21
4 (Figure 4) (The fastest train in Japan is Shinkanse
n.) is generated and output to the control unit 11 (FIG. 2).

In the control section 11, the number of characters of the text 34 supplied from the application 14, the number of commas (,), or the number of periods (.) Contained in the text 34 are counted, respectively, and as a teacher displayed on the CRT 3. The display time of the animation 32 (time to move the mouth of the animation 32) is calculated and supplied to the animation display unit 13.

In the animation display section 13, supplied from the application 14 via the control section 11,
The cell frame data of the animation 32 corresponding to the movement of the mouth for uttering the text 34, which is the response of the text 33, is read from the cell frame data storage unit 13a.

On the other hand, the voice synthesizing unit 12 calculates a parameter for synthesizing the voice corresponding to the text 34 which is the reply of the text 33 and is supplied from the application 14 via the control unit 11.

Then, when the mouse 6 is operated and the speak button 35 is clicked, in the control section 11, the start command is issued to the voice synthesis section 12 and the animation display section 1.
3 is output.

In the voice synthesizer 12, a parameter for synthesizing the voice corresponding to the text 34 is output to the voice synthesizer 1, and the voice synthesizer 1 uses the parameter "Voice The fastest train in Japan is Shinkanse".
n. "is synthesized and output from the built-in speaker (Fig. 5).

At the same time, in the animation display unit 13, the cell frame data of the animation 32 corresponding to the movement of the mouth for uttering the text 34 read out from the cell frame data storage unit 13a is stored in the control unit 11.
Corresponding to the display time of animation 32 (time to move the mouth of animation 32) supplied by CRT3
Are sequentially output to (Fig. 6), and an animation 3 in which the mouth is moved as if the text 34 is being spoken on the CRT 3
2 is displayed.

At this time, the text 34 (FIG. 4) is displayed on the window of the CRT 3 controlled by the X window system.

As described above, the voice synthesized from the text and the animation that moves corresponding to the text are output in synchronization.

Next, the operation will be further described with reference to FIG. First, in step S1,
It is determined whether or not text has been input from the keyboard 5. When it is determined in step S1 that the text is not input from the keyboard 5, the process returns to step S1 again. If it is determined in step S1 that text has been input from the keyboard 5, step S2
The cell frame data of the animation for moving the mouth corresponding to the text is read from the cell frame data storage unit 13a, and the time for outputting the cell frame data to the CRT 3, that is, the time for moving the mouth of the animation It is calculated, and the process proceeds to step S3.

In step S3, parameters for voice synthesis from the text input in step S1 are calculated, and the process proceeds to step S4. In step S4, a synchronization signal (for synchronizing the output timing of the voice synthesized from the parameters calculated in step S3 and the output timing of the cell frame data read from the cell frame data storage unit 13a in step S2 ( It is determined whether or not a start command) has been detected. When it is determined in step S4 that the sync signal is not detected, the process returns to step S4 again. When it is determined in step S4 that the synchronization signal is detected, the process proceeds to step S5, the voice synthesized from the parameters calculated in step S3 is output from the speaker incorporated in the voice synthesizer 1, and step S1 is performed. The animation of moving the mouth as if uttering the text input in step S2 is displayed on the CRT 3 in correspondence with the time of moving the mouth of the animation calculated in step S2.

Next, a method of calculating the animation display time (animation moving time) will be described. First, in order to display the cell frame (FIG. 8) stored in the cell frame data storage unit 13a like an animation, the number of times one cell frame is drawn by the CRT 3 is T. It should be noted that the number of times T is CRT3.
Is set in the EWS 4 in advance based on the display speed and the size of the image data.

Assuming that the number of cell frames displayed on the CRT 3 is N, the number of times TIMES that can display a cell frame that takes time (T × N) within the time when the synthesized voice is output is: It is calculated according to the following equation, where M is the number of characters in the text. TIMES = M / a However, a takes a variable value depending on whether or not a comma (,) or period (.) Is included in the text and the number of the commas. Is set in the EWS 4 in advance, similar to T described above.

Here, FIG. 9 is a diagram showing the relationship between the number of characters in the text and the utterance time. Although there are some variations depending on the words used in the text and the connections between the words and their surroundings, they fit within the rectangular area as shown in the figure. (According to this statistical result, the utterance time per character is 0.05
It is about a second. )

The reason why the above-mentioned a takes a variable value is, for example, the text "No. Japan is not a large country."
If there is a period or comma in the text, such as the text "Yes, I do.", Take a breath and a pause immediately after the utterance, so consider that the total utterance time may be slightly longer. This is because

FIG. 10 is a diagram showing a result of measuring the speech time of the text from the speech synthesizer 1. Figure 10 (a)
Is the waveform of the text "No. Japan is not a large country.", And Fig. 10 (b) is the text "No Japan is not a large country."
FIG. 10C shows a waveform of the text "Now Japan is not a large country." Comparing these three, the text "No. Japan is not a large countr" with a period in the text
It turns out that the utterance time of "y." is the longest.

By the method described above, the synchronization of the voice synthesized from the text and the animation that moves corresponding to the text can be realized within the audiovisually permissible range.

Further, when the voice synthesizer 1 finishes outputting the voice, the end signal is returned to the EWS 4 so that the EWS 4 outputs.
If the display of the moving image is ended when the end signal is detected in 4, the voice synthesized from the text and the animation that moves in response to the text are
Can be better synchronized.

If the word dictionary (FIG. 11) in which the utterance time of a word is defined is recorded in advance in, for example, a hard disk (not shown) built in the EWS 4, for example, the text “My uncle lives” can be recorded. in a small
When a text such as "town." is given, the words "my", "uncle", "live", "i" that compose the text are given.
The utterance times of "n", "a", "small", and "town" are respectively 0.148 seconds, 0.298 seconds, 0.2 from the word dictionary.
75 seconds, 0.103 seconds, 0.056 seconds, 0.319 seconds,
It is calculated as 0.231 seconds, and this total time (1.92
The animation can be displayed only for 8 seconds.

Furthermore, the correspondence table (FIG. 12) between the cell frame representing the mouth movement shown in FIG. 8 and the English phonetic symbols is shown in EW.
If it is recorded in advance on the hard disk built into S4, the text will be converted into phonetic symbols (see Fig. 1).
3) The animation can be generated by obtaining the cell frame to be displayed from the correspondence table.

As described above, since the voice synthesized from the text is output as if the animation moving corresponding to the text is output, the learner's interest is attracted and the learning effect is improved. You can

[0049]

According to the display control device of the first aspect, the voice synthesized from the text and the moving image displayed corresponding to the text are output in synchronization. Therefore, it is possible to output any sound in synchronization with the moving image.

According to the display control device of the second aspect,
Since the animation in which the mouth moves corresponding to the text is displayed on the display means, by applying it to a learning device such as a CAI system, the interest of the learner can be attracted and the learning effect can be improved. it can.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of a CAI system to which a display control device of the present invention is applied.

FIG. 2 is a subsystem “LAB” of the CAI system of FIG.
It is a figure which shows the user interface in "ORATORY."

3 is a more detailed block diagram of the application 14 of FIG.

4 is a diagram showing a screen displayed on a CRT 3 of the CAI system of FIG.

5 is a diagram for explaining input / output of the speech synthesizer 12 of FIG.

FIG. 6 is a diagram for explaining input / output of the animation display unit 13 of FIG.

FIG. 7 is a flowchart for explaining the operation of the interface of FIG.

8 is a diagram showing cell frame data stored in a cell frame data storage unit 13a of the animation display unit 13. FIG.

FIG. 9 is a diagram showing a result of measuring the relationship between the number of characters in a text and the utterance time of the text.

FIG. 10 is a diagram showing utterance times of text including commas or periods and text not including commas.

FIG. 11 is a diagram showing a word dictionary in which words are associated with utterance times of the words.

12 is a diagram showing a correspondence table between the cell frame data of FIG. 8 and phonetic symbols.

FIG. 13 is a diagram showing that text has been converted into phonetic symbols.

[Explanation of Codes] 1 voice synthesizer 2 terminal 3 CRT 4 workstation (EWS) 5 keyboard 6 mouse 11 control unit 12 voice synthesis unit 13 animation display unit 13a cell frame data storage unit 14 application 21 dialogue processing unit 22 language processing unit 23 Knowledge Database Search Section 24 Knowledge Database 31, 32 Animation 33, 34 Text 35 Speak Button

Front page continued (72) Inventor Fumitaka Kawate 6-735 Kitashinagawa, Shinagawa-ku, Tokyo Sony Corporation

Claims (2)

[Claims] 1. A voice synthesizing unit for synthesizing and outputting a voice from a text, a display unit for displaying a moving image corresponding to the text, a voice synthesized by the voice synthesizing unit, and a display by the display unit. And a synchronization means for synchronizing the captured moving image. 2. The display control device according to claim 1, wherein the display unit displays an animation in which a mouth moves corresponding to the text.