JP3413384B2 - Articulation state estimation display method and computer-readable recording medium recording computer program for the method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system and method for estimating the shape of a speaker's articulatory organ from a human voice waveform, and more particularly, using a physiological three-dimensional articulatory model, based on the voice waveform, the articulatory organ articulation. Systems and methods for visualizing and displaying status.

[0002]

2. Description of the Related Art Learning a foreign language, especially learning a conversation in a foreign language, is considered to be difficult for a Japanese person. There are various possible causes, but it is considered that the main cause is that the speaker does not understand the articulation state of his or her own speech organs and their adjustment. In particular, the pronunciation of a foreign language must be controlled appropriately in order to imitate the pronunciation of a teacher, and to control the articulatory organs of the tongue and lower jaw.
Such learning work is extremely difficult for the average Japanese who rarely have the opportunity to listen to and speak foreign languages on a daily basis.

After all, these difficulties are related to how to adjust one's articulatory organs to produce a correct pronunciation.
This is because it is not possible to give feedback because there is no teacher. As a result, they practice their own pronunciation but cannot improve their pronunciation at all. Even if there is a teacher, it is actually not so easy to control the articulatory organs so that the teacher hears the pronunciation and can produce a sound similar to the voice. In the first place, it is not easy to know even what shape the articulatory organ of one's own voice is. Therefore, in the long run, it may not be possible to expect improvement in a foreign language.

These problems are not limited to learning a foreign language. For example, a person who has some difficulty in controlling articulatory organs, or someone who cannot reliably grasp the relationship between his or her voice and articulatory organs due to hearing impairment may use the voice to In order to communicate, it is desirable to train articulatory control so that normal pronunciation is possible. However, even in this case, it is not easy to know how to control the articulatory organs of oneself to produce the desired pronunciation, and as a result, it is difficult to perform the appropriate utterance.

In order to solve these problems, it is not enough to teach the voice and the corresponding articulation state and train the learner to achieve the same articulation state. It is desirable to give feedback to the articulatory state by recognizing the difference between the learner's articulatory state at the time of learning and the model. For that purpose, it is necessary to estimate the articulatory state of the learner from the voice.

At present, the methods for estimating articulatory states from speech waves are roughly divided into those that estimate an acoustic tube equivalent to acoustic features of speech using an idealized vocal tract model (Shroeder, MR : "Determination of the geometry of
the human vocal tract by acoustic measurement, "
J. Acoust. Soc. Am., 4, p. 1002 (1967), Yehia, H.
and Itakura, F: "A method to combine acoustic and
morphological constrain t s in the speech production
inverse problem, "Speech Comm. 18, 151-174 (199
6)), and one that estimates an articulatory target using an articulatory model with a stronger constraint (Shirai, Honda: "Estimation of articulatory parameters from speech waves," IEICE, 61, 409-416 (19).
78)).

[0007]

However, the method of estimating the articulation state from the voice wave has a problem that the correspondence relationship between the voice wave and the articulation state is not unique. Various constraint conditions have been considered in order to solve the inverse problem, but the accuracy of the estimation result is low and the reliability is not high.

Therefore, an object of the present invention is to provide a method and computer for estimating and displaying an articulatory state capable of estimating an articulatory state when a voice is emitted from a sound wave and presenting the articulated state in a format easier to understand than conventional methods. A computer-readable recording medium that stores a computer program that operates as an articulatory state display device.

[0009]

According to one aspect of the present invention, there is provided a method for estimating and displaying an articulatory state for estimating and displaying an articulatory state of a speaker's articulatory organ from an input audio signal. Extracting a predetermined parameter,
A step of setting an initial articulation target based on the extracted predetermined parameter; a step of estimating the articulation state of the speaker's speech organ corresponding to the input voice signal based on the articulation target and the predetermined parameter; and including. The estimating step includes the step of estimating the articulatory state corresponding to the input voice signal by using the physiological three-dimensional articulatory model based on the articulatory target, and the synthetic voice signal by the predetermined acoustic model based on the estimated articulatory state. And a step of extracting a predetermined parameter from the synthesized voice signal, and a predetermined parameter extracted from the synthesized voice signal so as to approach a predetermined relationship with the predetermined parameter extracted from the input voice signal. Updating the articulation target. The method for estimating and displaying the articulatory state further includes a step of determining whether or not the predetermined parameter extracted from the synthesized voice signal satisfies a predetermined relationship with the predetermined parameter extracted from the input voice signal, and a step of determining Display device for displaying the articulation state of the articulatory organ obtained by driving the physiological three-dimensional articulation model according to the articulation target. And the step of selectively performing the processing displayed above.

According to this method, the articulation state of the articulatory organ at the time of utterance can be visually displayed by estimating the articulation state from the voice wave. It becomes easy for the speaker or the like to visually check the articulatory state and adapt it to the correct articulatory state.

The method according to the present invention preferably further includes, for each frame included in the stream of the input audio signal, performing steps from extracting a predetermined parameter from the input audio signal to selectively performing the parameter. Includes the step of animating the articulation state.

Since the articulatory state is displayed for each frame included in the stream of the input audio signal, the articulatory state of the articulatory organ at the time of utterance is displayed as an animation for easier understanding and dynamic articulatory state. It becomes easy to learn utterances that need to be changed.

Preferably, the step of displaying includes the step of three-dimensionally displaying the shape of the articulatory organ on the display device.

The articulated state can be intuitively recognized by the three-dimensional display. More preferably, the step of displaying three-dimensionally includes the step of displaying the three-dimensional shape of the articulatory organ on the display device by arranging a plurality of two-dimensional cross-sectional shapes of the articulatory organ.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION In order to improve the accuracy and reliability of the estimation result of the articulation state as described above, it is desirable to use an articulation model faithfully based on the human speech mechanism.
In connection with this, three-dimensional MRI (magnetic r
esonance imaging) A method of constructing a three-dimensional physiological articulatory model based on image data has been proposed (Party, Honda: "Synthesis of vowel sequences using physiological articulatory model," Sound lecture, (1998, 9 ), Dang , J and Honda, K., "Speech pro
duction of vowel sequences using a physiologicalar
ticulatory model, "Proc. ICSLP98, Vol. 5, pp.1767-
1770 (1998), Party, Honda: "Synthesis method using physiological articulatory model," Onkyo, 243-244 (1999, 9)). This model has physiological constraints related to the structure of the main speech organs and the structure of the muscles around it based on the physiological characteristics of the speech organs (articulatory organs) and the muscles around them. By describing the relationship between the goal and the contractile force of muscles, the human speech mechanism can be reproduced almost faithfully. Therefore, by applying this physiological articulatory model, it is possible to obtain a more natural vocal tract shape from the sound wave.

The system according to the first embodiment of the present invention described below uses the physiological articulatory model, estimates the articulatory state from the voice wave in real time, and displays the animation, so that the speaker can easily perform self-expression. It is for understanding the articulatory state of.
This system can be used as an auxiliary device for learning foreign languages, learning pronunciation of children or people with disabilities. [Hardware Configuration] Hereinafter, an articulation state display device for implementing the method according to the first embodiment of the present invention will be described. The articulatory state display device, a personal computer or workstation or the like, and the computer, there is implemented by software executing on the computer, from the uttered speech waveform human estimated articulatory state of a speaker In addition to displaying an animation, a synthetic voice is produced based on the estimated articulation state. FIG. 1 shows the appearance of this articulation state display device.

Referring to FIG. 1, the articulated state estimation display device 20 is a CD-ROM (Compact Disc Read-Only Memor).
y) A computer main body 40 having a drive 50 and an FD (Flexible Disk) drive 52, and a display 42 as a display device connected to the computer main body 40.
A keyboard 46 and a mouse 48, which are also connected to the computer main body 40 as input devices, a microphone 30 connected to the computer main body 40 for capturing a voice uttered by a speaker, and for outputting a synthetic voice. And a speaker 32 having a built-in amplifier.

FIG. 2 is a block diagram showing the structure of the articulated state estimation display device 20. As shown in FIG. 2, the computer main body 40 constituting the articulated state estimation display device 20 includes a CPU connected to a bus 66 in addition to a CD-ROM drive 50 and an FD drive 52.
(Central Processing Unit) 56 and ROM (ReadOnl
y Memory) 58 and RAM (Random Access Memory) 6
0, a hard disk 54, a voice capturing device 68 for capturing a voice from the microphone 30, and a sound with a built-in sound source for generating a signal for driving the speaker 32 from a sound signal provided from the CPU 56. It includes a board 70. The CD-ROM drive 50 has a CD-RO
M62 is installed. An FD 64 is attached to the FD drive 52.

As described above, the main part of this articulation state display device is realized by computer hardware and software executed by the CPU 56. Generally, such software is CD-ROM 62, FD6.
It is stored in a storage medium such as No. 4 and distributed, and is read from the storage medium by the CD-ROM drive 50 or the FD drive 52 and stored in the hard disk 54 once. Alternatively, when the device is connected to the network, the hard disk 5 is temporarily sent from the server on the network.
4 is copied to. Then further hard disk 54
Is read from the RAM 60 to the RAM 60 and executed by the CPU 56. When connected to the network, it may be directly loaded into the RAM 60 and executed without being stored in the hard disk 54.

The hardware itself of the computer shown in FIGS. 1 and 2 and the operating principle thereof are general. Therefore, the most essential part of the present invention is the software stored in the storage medium such as the FD drive 52, the FD 64, and the hard disk 54.

As a recent general tendency, various program modules are prepared as a part of the operating system of a computer, and an application program mainly calls these modules in a predetermined arrangement when necessary to proceed with processing. May only play a role. In such a case, the software itself for realizing the articulatory state display device does not include such a functional module, and the hardware of the computer concerned and the operating system of the present invention are not used until this embodiment of the present embodiment is performed. The articulation state display device will be realized. However, as long as a general platform is used, it is not necessary to distribute software that includes such modules, and the software itself that does not include those modules and a recording medium that records these software (and those software are distributed on the network). The data signal in the case) can be considered to constitute an embodiment. [Functional Configuration] Referring to FIG. 3, the articulatory state estimation and display device 20 according to the first embodiment corrects the radiation characteristic with respect to the effect of radiation from the lips by pre-emphasising the actual voice. A radiation characteristic correction unit 80, an adaptive filter 82 and an autocorrelation calculation unit 84 for flattening the spectrum of the corrected voice, and an LPC as an acoustic parameter from the input voice signal with the flattened spectrum. L for calculating the cepstrum coefficient by calculation
A PC (linear predictive coding) cepstrum calculation unit 90 and an initial articulation target storage unit 100 for storing an LPC cepstrum obtained in advance for some speeches as an initial articulation target for iterative processing described later.
And the LPC cepstrum coefficient for the actual voice obtained by the LPC cepstrum calculation section 90 and the LPC cepstrum coefficients of some initial articulation targets stored in the initial articulation target storage section 100 are compared to obtain the actual voice. The LPC cepstrum coefficient closest to the LPC cepstrum coefficient is selected as an initial articulation target, and the articulation model is driven by starting from the initial articulation target set by the initial target setting unit 96. And updating the articulation target while performing iterative processing so that the difference between the LPC cepstrum coefficient calculated from the synthetic speech obtained from the updated articulation state and the LPC cepstrum coefficient calculated by the LPC cepstrum calculation unit 90 becomes small. When the actual articulation given at the beginning is spoken, As articulation state of a speaker, and a articulatory model iterative calculation unit 98 for displaying (108) the shape of the articulator animation from the resulting articulation state.

The articulation model iterative calculation unit 98 estimates the contraction pattern of the articulatory organ and the muscles around it using the articulation target initially set by the initial target setting unit 96, and drives the articulation model by the contraction force of the muscles. And a synthetic voice based on the vocal tract shape generated by the articulatory model unit 102 (11
0) acoustic model unit 104 and acoustic model unit 1
04, an adaptive filter 86 and an autocorrelation calculation unit 88 for flattening the spectrum by performing the same processing as that performed by the adaptive filter 82 and the autocorrelation calculation unit 84, and the spectrum LPC cepstrum calculator 90 from the flattened synthesized speech of
The LPC cepstrum calculation unit 92 for calculating the LPC cepstrum coefficient as an acoustic parameter in the same manner as
And the LPC cepstrum coefficient output by the LPC cepstrum calculation section 90 and the LPC cepstrum coefficient output by the LPC cepstrum calculation section 92 are close to each other, that is, the difference between the two is small, and the articulation model section 10
Articulatory target updating unit 94 for estimating and updating the articulatory target for 2).

Referring to FIG. 4, the acoustic model section 104 is
Output from the cross-sectional area function estimating unit 126 for estimating the vocal tract cross-sectional area shape based on the three cross-sectional shapes 124 of the articulatory organ region output by the articulatory model unit 102, and the cross-sectional area function estimating unit 126. A sound source is generated by driving an electric circuit model by changing parameters based on the vocal tract cross-sectional area shape.
And an electric circuit model unit 128 for generating a synthesized voice. [Control Structure of Software] With reference to FIG. 5, the software for realizing the articulated state estimation display device 20 shown in FIGS. 3 and 4 has the following control structure. In the example described below, it is assumed that the radiation characteristics and the spectrum flattening processing have already been performed on the input voice. Note that, in addition to the correction of the radiation characteristic and the flattening process of the spectrum, the calculation of the LPC cepstrum coefficient, the synthesis of the voice, the animation display of the vocal tract state based on the articulatory model are all performed by software in the present embodiment. Any of these may be performed using a dedicated LSI circuit (Large Scale Integrated circuit). The processing described below is performed for each frame of the stream of the input audio signal. Further, although the expression of each frame is used in the following processing, in short, the following processing may be applied to a predetermined unit of the input audio signal which is a target of audio processing.

First, an initial LP is applied to the input real voice.
The C cepstrum coefficient is calculated (140). This corresponds to the processing in the LPC cepstrum calculation unit 90. An initial articulation target having an acoustic parameter closest to the calculated LPC cepstrum coefficient is selected from candidates of the initial articulation target stored in the initial articulation target storage unit 100 (142). In the present embodiment, the standard articulation target of five Japanese vowels is prepared in advance as the initial articulation target, and one of these vowels that is closest to the first input actual voice is selected as the initial articulation target.

Subsequently, using this articulatory target, the contraction pattern of the muscle of the articulatory organ is estimated, and the articulatory model is driven according to the obtained contractile force of the muscle (144) to generate a new vocal tract shape (146). . Using this vocal tract shape, a synthetic voice is output (110) by an acoustic model.

For the synthesized speech thus obtained,
The same spectral flattening process that was performed before step 140 is performed, and the same as step 140,
The LPC cepstrum coefficient is calculated (148).

The acoustic parameters of the synthetic voice and the input voice are compared, and it is determined whether the difference between them is within a predetermined range (150). If the difference is within the predetermined range, it means that the articulation target has approached an almost constant target as a result of the iteration, and it is meaningless to perform the process any more. Therefore, the processing ends here, and the articulation state is displayed as an image based on the articulation model obtained at this time (154). As this image, the articulation model unit 102
It is possible to use the three two-dimensional cross-sectional shapes 124 output by, and by arranging these three-dimensionally, the estimated shape of the articulatory organ of the speaker is visually and intuitively arranged as shown in FIG. It can be displayed in an easy-to-understand manner.

When it is determined that the difference between the synthesized voice and the input voice is large as a result of the determination in step 150, the acoustic parameter obtained here is articulated so as to be close to the acoustic parameter obtained from the actual voice. The target is updated (152) and control returns to step 144. Hereinafter, by repeating the processes in and after step 144, finally articulation goal as constant, i.e., the actual speaker tone sound
An affirmative determination is obtained in step 152 by obtaining an articulation target that drives the articulation model so that an articulation state that closely matches

When the display of the articulated state ends, the control returns to step 140, and the above-mentioned processing is performed again for the voice of the next frame. By repeating this processing for the stream of the input audio signal, the shape of the articulatory organ at the time of the speaker's utterance is displayed on the display 42 in an animation manner. By comparing this articulatory organ shape with an exemplary articulatory organ shape animation displayed on another window, the speaker can give very effective feedback to his articulatory movement. It will be possible.

[0030] The animation display articulators based on articulatory model in a system of the embodiment described above, but is performed by arranging three sagittal cross section, limited to three is the number of sagittal cross section It is not done. An appropriate number can be selected depending on the performance of the computer and peripherals and the conditions required by the application.
Further, the output from the articulatory model unit 102 is not limited to a three-dimensional display by simply displaying a plurality of sagittal cross-sections side by side, but three-dimensional shape data representing the surface of the articulatory organ (inner surface of vocal tract). Alternatively, the three-dimensional shape may be represented. Also,
In some cases, the three-dimensional shape is not always necessary, and in that case, one sagittal cross-sectional shape may be displayed, or a plurality of sagittal cross-sectional shapes may be displayed separately.

Although the system of the above-described embodiment is a stand-alone system, the animation display can be transmitted to the computer at another point by the network and can be displayed there. This may be unidirectional or bidirectional. For example, in the case of interactive animation display, when a language teacher teaches a learner in a remote place, the teacher confirms the learner's vocal tract shape and gives more appropriate instructions to correct it. it it becomes possible to carry out, the learner is self-regulating by looking at the teacher of the vocal tract shape
It will be possible to control sound organs more appropriately.

When connecting computers by such a network, the connection is not limited to one-to-one connection. If there is more than one learner for each teacher, by broadcasting the vocal tract shape animation obtained from the teacher's voice to those learners,
It can be expected that such a language learning system will become more efficient.

Further, although the system of the embodiment disclosed this time displays the vocal tract shape as an animation in real time, it does not handle the contents of voice. However, from the vocal tract shape, the constraint condition is set so that the voice uttered at that time can be estimated accurately.
Furthermore, by capturing such changes in vocal tract shape over time, this technique may be applicable to speech recognition.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

[Brief description of drawings]

FIG. 1 is an external view of a system according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a hardware configuration of the system according to the first embodiment of the present invention.

FIG. 3 is a functional block diagram of the system according to the first embodiment of the present invention.

FIG. 4 is a block diagram of a voice synthesis processing unit in the system according to the first exemplary embodiment of the present invention.

FIG. 5 is a flowchart showing an outline of processing performed by the system according to the first embodiment of the present invention.

FIG. 6 is a diagram showing an example of an articulatory model displayed in animation.

[Explanation of symbols]

20 articulation state display device, 30 microphone, 32 speaker, 40 computer main body, 42 display, 8
2,86 adaptive filter, 84,88 autocorrelation calculation unit, 90,92 LPC cepstrum calculation unit, 94 articulation target update unit, 96 initial target setting unit, 98 articulation model iterative calculation unit, 102 articulation model unit, 104 acoustic model unit .

Continuation of front page (56) References JP-A-6-348297 (JP, A) JP-A-10-254497 (JP, A) JP-A-59-71096 (JP, A) JP-A-59-124385 (JP , A) JP 11-259097 (JP, A) JP 62-184500 (JP, A) JP 62-184499 (JP, A) JP 11-202897 (JP, A) Kiyoshi Honda, Generation of three-dimensional vocal tract shape based on physiological articulatory model, Proceedings of Acoustical Society of Japan, Acoustical Society of Japan, March 1998, Spring 1998, I, 265-266 Atsushi Sugiura, Masafumi Matsumura, Magnetics Measurement of 3D vocal tract shape by resonance imaging, IEICE technical report, IEICE, 26 January 1990, SP89-109 ~ 118, 65-72 Yu Sasaki, et al., Speech Interactive expression of three-dimensional vocal tract shape for education, Proceedings of Acoustical Society of Japan, Acoustical Society of Japan, March 1998, Spring 1998, 341-342 (58) Fields investigated (Int.Cl. ⁷ , DB name) G10L 13/00

Claims (8)

(57) [Claims] 1. A method for estimating and displaying an articulatory state of a speaker's articulatory organ from an input audio signal, comprising: extracting a predetermined parameter from the input audio signal; And a step of setting an initial articulation target based on the predetermined parameter, based on the articulation target and the predetermined parameter,
Estimating the articulatory state of the speaker's speech organ corresponding to the input voice signal, wherein the estimating step uses the physiological three-dimensional articulatory model to determine the input voice signal based on the articulation target. Estimating a corresponding articulation state, generating a synthesized voice signal by a predetermined acoustic model based on the estimated articulation state, extracting the predetermined parameter from the synthesized voice signal, and Updating the articulation target such that the predetermined parameter extracted from the synthesized voice signal approaches a predetermined relationship with the predetermined parameter extracted from the input voice signal, and the estimated display of the articulation state is included. The method further comprises the step of: the predetermined parameter extracted from the synthesized audio signal being extracted from the input audio signal. A step of determining whether or not the predetermined parameter and the predetermined relationship are satisfied, and a step of repeating the process again from the step of estimating based on the updated articulation target according to the determination result of the determining step, and And selectively displaying the articulatory state of the articulatory organ obtained by driving the physiological three-dimensional articulatory model according to the articulatory target on a display device. 2. By further performing the steps from the step of extracting a predetermined parameter from the input audio signal to the step of selectively performing each of the frames included in the stream of the input audio signal. The method for estimating and displaying the articulatory state according to claim 1, comprising a step of displaying the articulatory state as an animation. 3. The method for estimating and displaying an articulatory state according to claim 1, wherein the displaying step includes a step of three-dimensionally displaying the shape of the articulatory organ on the display device. 4. The three-dimensionally displaying step includes a step of displaying a three-dimensional shape of the articulatory organ on the display device by arranging a plurality of two-dimensional cross-sectional shapes of the articulatory organ. The method for estimating and displaying the articulatory state according to claim 3. 5. A computer-readable recording medium storing a computer program for causing a computer to execute an articulation state estimation display method for estimating and displaying an articulation state of a speaker's articulatory organ from an input voice signal. Therein, the method comprises a step of extracting a predetermined parameter from an input audio signal, a step of setting an initial articulation target based on the extracted predetermined parameter, the articulation target, and the predetermined parameter. On the basis of,
Estimating the articulatory state of the speaker's speech organ corresponding to the input voice signal, wherein the estimating step uses the physiological three-dimensional articulatory model to determine the input voice signal based on the articulation target. Estimating a corresponding articulation state, generating a synthesized voice signal by a predetermined acoustic model based on the estimated articulation state, extracting the predetermined parameter from the synthesized voice signal, and Updating the articulation target such that the predetermined parameter extracted from the synthesized voice signal approaches a predetermined relationship with the predetermined parameter extracted from the input voice signal, the method further comprising: The predetermined parameter extracted from the synthesized voice signal is combined with the predetermined parameter extracted from the input voice signal. A step of determining whether or not a predetermined relationship is satisfied, a process of repeating the process again from the estimating step based on the updated articulation target according to the determination result of the determination step, and the articulation target. And a step of selectively displaying, on a display device, an articulatory state of an articulatory organ obtained by driving a physiological three-dimensional articulatory model in accordance with the method. 6. The method further includes, for each frame included in the stream of the input audio signal, from the step of extracting a predetermined parameter from the input audio signal to the step of selectively performing the parameter. The computer-readable recording medium according to claim 5, further comprising the step of animating the articulation state. 7. The computer-readable recording medium according to claim 5, wherein the displaying step includes a step of three-dimensionally displaying the shape of the articulatory organ on the display device. 8. The three-dimensionally displaying step includes a step of displaying a three-dimensional shape of the articulatory organ on the display device by arranging a plurality of cross-sectional shapes of the articulatory organ. Item 7. A computer-readable recording medium in the articulated state according to Item 7.