JP4543919B2 - Language learning device - Google Patents

Description

  The present invention relates to a language learning system that supports language learning.

  In language learning of foreign languages or native languages, especially self-study of pronunciation or utterance, a model voice recorded on a recording medium such as a CD (Compact Disk) is played, and the model voice is imitated to pronounce or speak. The learning method is widely used. The purpose of this is to acquire correct pronunciation by imitating model voices. Here, in order to advance learning more effectively, it is necessary to objectively evaluate the difference between the model voice and one's own voice. However, there is a problem that it is difficult to specifically grasp the difference between the voice produced by the user and the model voice only by listening to the model voice recorded on the CD and imitating the model voice.

As a technique for solving such a problem, there are techniques described in Patent Documents 1 to 3, for example. Patent Document 1 discloses a technique for simultaneously reproducing an exemplary voice and a user voice. Patent Document 2 discloses a technique for simultaneously outputting a model voice waveform (model waveform) and a user voice waveform (voice waveform). Japanese Patent Application Laid-Open No. 2004-151867 discloses a technique for uniformly expanding one voice in order to align both heads when comparing the model voice and the user voice, or to make the lengths of both the same.
JP 7-219418 A JP 2002-23613 A Japanese Patent Laid-Open No. 11-14396

  The technique described in Patent Document 1 has a problem that only the model voice and the user voice are reproduced at the same time, and the difference between the two is difficult to understand. The technique described in Patent Document 2 also has a problem in that only the waveform of the model voice and the waveform of the user voice are output at the same time, and the difference between the two is difficult to understand. Furthermore, in the technique described in Patent Document 3, in order to align the lengths of the model voice and the user voice, only the length of one voice is stretched uniformly, and the two phonemes do not necessarily match the time axis. There was a problem of not doing.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a language learning device capable of presenting differences between a model voice and a user voice.

  In order to solve the above-described problem, the present invention provides a storage unit that stores an exemplary voice, an input unit that inputs a user voice, and a separation that separates the exemplary voice and the user voice into a consonant part and a vowel part, respectively. And the processing unit for changing the length of the vowel part of the exemplary voice or the user voice so that the lengths of the exemplary voice and the user voice are the same. Provided is a language learning device having comparison means for comparing a model voice and a user voice and extracting a difference point.

In a preferred embodiment, the language learning device further includes a rearrangement unit that rearranges the consonant part of either the model voice or the user voice so that the position on the time axis is the same as that of the other consonant part. And the processing means changes a length of a vowel part positioned between a consonant part rearranged by the rearrangement means and a consonant part appearing next to the consonant part on the time axis. Thus, the model voice and the user voice may have the same length.
In another preferred embodiment, the language learning device extracts predetermined parameters for the user voice and the model voice, and generates image data indicating a graphic corresponding to a change amount of the parameter; You may further have a display means to display the image data produced | generated by the image production | generation means in a different display mode by the difference extracted by the said comparison means.

In still another preferred embodiment, the language learning device includes a difference point emphasizing unit that emphasizes the difference points extracted by the comparison unit, and an output unit that outputs a voice in which the difference points are emphasized by the difference point emphasizing unit. May further be included.
In this aspect, the difference highlighting means makes the volume of the model voice higher than the volume of the user voice for the part where the difference is extracted by the comparison means, and the part where the difference is not extracted by the comparison means The volume of the model voice may be made smaller than the volume of the user voice.
Alternatively, the difference point emphasizing unit may amplify a specific frequency region in the model voice for the portion from which the difference point is extracted by the comparison unit.

  According to the present invention, the user can specifically specify a portion where there is a difference between the user's voice and the model voice.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<First Embodiment>
FIG. 1 is a block diagram showing a functional configuration of a language learning device 1 according to the first embodiment of the present invention. The memory | storage part 10 memorize | stores model audio | voice data which shows the audio | voice as a model in language learning. The input unit 20 acquires the voice of the user (learner or student) and outputs user voice data. The data processing unit 30 processes the user voice data so that the user voice and the model voice have the same length. The difference point extraction unit 40 compares the user voice data and the model voice data that are arranged to have the same length, and extracts the difference between the two. The difference point emphasizing unit 50 performs processing for emphasizing the difference points extracted by the difference point extracting unit 40 with respect to the model voice data and the user voice data. The sound output unit 60 reproduces the model sound and the user sound in which the difference is emphasized. Details of the function of each component will be described later.

  FIG. 2 is a block diagram illustrating a hardware configuration of the language learning device 1. A CPU (Central Processing Unit) 101 reads and executes a program stored in a ROM (Read Only Memory) 103 or an HDD (Hard Disk Drive) 104 using a RAM (Random Access Memory) 102 as a work area. The HDD 104 is a storage device that stores various application programs and data. Regarding the present embodiment, in particular, the HDD 104 stores a language learning program and a model voice database DB1 in which model voice data used in the language learning program is recorded.

  The display 105 is a display device that displays characters and images under the control of the CPU 101, such as a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display). The microphone 106 is a sound collection device for acquiring the user's voice, and outputs a voice signal corresponding to the voice uttered by the user. The sound processing unit 107 has a function of converting an analog sound signal output from the microphone 106 into digital sound data, and a function of converting sound data stored in the HDD 104 into a sound signal and outputting the sound signal to the speaker 108. Further, the user can input an instruction to the language learning device 1 by operating the keyboard 109. Each component described above is connected to each other via a bus 110.

  FIG. 3 is a diagram showing the contents of the model voice database DB1. In the exemplary speech database DB1, text data of example sentences used for language learning (for example, “Good to see you again. How are you?” Etc. in English learning) and voices obtained by digitizing speech waveforms in units of example sentences. A plurality of waveform data and identifiers specifying the example sentences are stored. Example sentence text data, speech waveform data, and an identifier are associated with each other.

  Next, the operation of the language learning device 1 will be described. In the present embodiment, when the CPU 101 executes a language learning program stored in the HDD 104, functions corresponding to the functional components shown in FIG.

  FIG. 4 is a flowchart showing the operation of the language learning device 1 according to the present embodiment. When the language learning program is executed, the CPU 101 displays a message prompting the user to select an example sentence on the display 105. In accordance with the message displayed on the display 105, the user selects one example sentence from the example sentences recorded in the model voice database DB1. The CPU 101 reproduces the voice of the selected example sentence (step S101). Specifically, it is as follows. The CPU 101 extracts model voice data corresponding to the selected example sentence from the model voice database DB1. The CPU 101 outputs the extracted model voice data to the voice processing unit 107. The voice processing unit 107 performs digital / analog conversion on the input model voice data and outputs the analog voice signal to the speaker 108. In this way, the model voice is reproduced.

  The user listens to the model voice reproduced from the speaker 108, and utters an example sentence simulating the model voice toward the microphone 106. That is, a user voice is input (step S102). Specifically, it is as follows. When the reproduction of the model voice is finished, the CPU 101 displays a message on the display 105 urging the user to generate an example sentence such as “Next is your turn. Please pronounce the example sentence”. Further, the CPU 101 displays a message on the display 105 instructing an operation for inputting the user voice, such as “Sound after pressing the space key and press the space key again when the sound is finished”. The user operates the keyboard 109 according to the message displayed on the display 105 and inputs the user voice. That is, after the space key on the keyboard 109 is pressed, the example sentence is uttered toward the microphone 106. When the utterance is finished, the user presses the space key again.

  The user's voice is converted into an electric signal by the microphone 106. The microphone 106 outputs a user voice signal. The user voice signal is converted into digital voice data by the voice processing unit 107 and recorded in the HDD 104 as user voice data. After the reproduction of the model voice is completed, the CPU 101 starts the recording of the user voice data with the press of the space key as a trigger, and ends the recording of the user voice data with the press of the space key again as a trigger. That is, the user voice from when the user first presses the space key until the user presses the space key again is recorded in the HDD 104.

  Subsequently, the CPU 101 processes the user voice data so that the user voice and the model voice have the same length (step S103). Specifically, it is as follows. FIG. 5 is a diagram illustrating waveforms of the model voice (FIG. 5A) and the user voice (FIG. 5B). In the example shown in FIG. 5, the model voice and the user voice are both uttered by the same example sentence, but have different lengths because the utterance speeds are different. That is, since the voice of the user voice is slower, the length of the voice is longer. The CPU 101 makes the length of the user voice the same as that of the model voice as follows.

  FIG. 6 is a flowchart showing in more detail the process of making the lengths of the user voice and the model voice the same in step S103. First, the CPU 101 calculates the lengths of the model voice and the user voice by a method such as measuring the data size (step S103-1). From this calculation result, the CPU 101 further calculates the difference in length between the model voice and the user voice (Δt in FIG. 5).

  Subsequently, the CPU 101 adds an identifier to each of the part related to the consonant and the part related to the vowel in the user voice data (step S103-2). This is for the following purpose. That is, the length of consonants is generally the same regardless of the speaker, whereas the length of vowels varies greatly depending on the speaker. Therefore, when changing the length of the user voice, if the length of the consonant is kept unchanged and only the length of the vowel is changed, the length of the voice can be changed without causing unnaturalness in hearing. it can. In the prior art, since the length of the voice is uniformly changed regardless of whether it is a consonant or a vowel, there is a problem that an unnatural audibility is given. However, according to the present embodiment, such a problem does not occur.

  For example, the vowel part and the consonant part are separated as follows. The CPU 101 extracts vowels included in the example sentence from the text data of the selected example sentence. For example, consider a case where vowels “a”, “u”, “i”, and “e” are extracted in order from the top of text data. The CPU 101 divides the audio data every predetermined time (frame). The CPU 101 obtains the logarithm of the amplitude spectrum obtained by Fourier transforming the waveform indicated by the exemplary voice data decomposed into frames and the waveform indicated by the user voice signal, and inverse Fourier transforms it to obtain a spectrum envelope for each frame. . The CPU 101 extracts the formant frequencies of the first formant and the second and third formants from the spectrum envelope thus obtained. In general, a vowel is characterized by a distribution of first, second, and third formants. The CPU 101 first matches the formant frequency distribution of the vowel “a” from the beginning of the audio data. If it is determined by matching that the frame corresponds to the vowel “a”, the CPU 101 generates data C indicating the type of the detected vowel and the position on the audio data. When the data C has already been generated, new information is added to the data C. The CPU 101 performs matching with the vowel “a” for the subsequent frames, and subsequently performs matching with the vowel “u” when there is no match. In this way, vowels are searched from the beginning, and data C is generated. Instead of generating data indicating the type and position of vowels, data indicating the position of consonants may be generated, or data indicating the positions of both vowels and consonants may be generated. Also, section position data of sound and sound liaison sections (intermediate sounds) or silence section data may be generated.

  Subsequently, the CPU 101 assigns numbers to the parts related to the consonant in the model voice data in order from the head of the data, together with information indicating the head position of each part related to the consonant (for example, the time from the head of the data). The data is stored in the RAM 102 as the table TB1. Here, the separation of the vowels and the consonants in the model voice data may be performed in the same manner as in the case of the user voice data described above, or information indicating the position of the vowel or the position of the consonant is stored in the model voice database DB1 in advance. Alternatively, the consonant part may be specified based on the information. Subsequently, the CPU 101 cuts out the part related to the consonant from the user voice data, and refers to the table TB1, and rearranges the cut out consonant part so that the positions of the model voice and the consonant match (step S103-3). Further, the CPU 101 processes the data so that the portion related to the vowel located between the consonants in the user voice data has the same length as the model voice (step S103-3). For example, when the vowel part of the user voice is longer than the vowel part of the model voice, this can be realized by deleting the data of the long part. Alternatively, if the vowel part of the user voice is shorter than the vowel part of the model voice, the waveforms of the vowel part may be repeatedly added until the desired length is reached. In this way, the length of the user voice is the same as that of the model voice, and the positions of consonants and vowels on the time axis also coincide.

  A description will be given with reference to FIG. 4 again. The CPU 101 extracts a difference between the model voice and the user voice (step S104). This process is performed as follows, for example. As described above, the CPU 101 divides the waveform indicated by the model voice data at predetermined time (frames). The CPU 101 also divides the waveform indicated by the user voice data for each frame. The CPU 101 obtains the logarithm of the amplitude spectrum obtained by Fourier transforming the waveform indicated by the exemplary voice data decomposed into frames and the waveform indicated by the user voice signal, and inverse Fourier transforms it to obtain a spectrum envelope for each frame. .

  FIG. 7 is a diagram illustrating a spectrum envelope of an exemplary voice (upper) and a user voice (lower). The spectrum envelope shown in FIG. 7 is composed of three frames, frame I to frame III. The CPU 101 compares the obtained spectrum envelope for each frame. When the difference between the spectrum envelope of the model voice and the spectrum envelope of the user voice exceeds a predetermined threshold, the CPU 101 determines that there is a difference between the model voice and the user voice in the frame. The difference between the model voice and the user voice may be obtained, for example, by a distance between two points when a characteristic formant frequency and a spectral density are represented in a spectral density-frequency diagram, or a spectrum at a specific frequency. You may obtain | require by comparing a density. Alternatively, the difference between the model voice and the user voice may be obtained by comparing the formant frequencies of one or more specific formants. In the example shown in FIG. 5, the CPU 101 determines that there is a difference with respect to the frame II. The CPU 101 generates data F in which a flag indicating that there is a difference between the model voice and the user voice is generated and stored in the RAM 102. If there is no difference between the model voice and the user voice, the CPU 101 generates data F indicating that and stores it in the RAM 102. Further, the CPU 101 generates data D in which a flag indicating whether or not there is a difference between the model voice and the user voice in the frame is generated and stored in the RAM 102. That is, the data D indicates whether the user's pronunciation is good or not for each frame (whether there is a difference from the model voice). In this way, the CPU 101 compares the spectrum of the model voice and the spectrum of the user voice for all frames. The RAM 102 stores data D that identifies a frame that is determined to be different from the model voice.

  A description will be given with reference to FIG. 4 again. Based on the data F, the CPU 101 determines whether there is a portion that is different from the model voice in the user voice (step S105). When there is a part that is different from the model voice in the user voice (step S105: YES), the CPU 101 performs a difference enhancement process described below (step S106). The CPU 101 performs the processing of steps S105 and S106 over all frames (step S108). Thereby, the model voice in which the difference is emphasized is reproduced. If there is no bad part in the user's pronunciation (step S105: NO), the CPU 101 displays a message such as “good” on the display 105 and ends the process.

  The difference point emphasis process in step S106 is performed as follows, for example. For the purpose of the user confirming his / her pronunciation, the user voice is basically reproduced in step S107 described later. However, with respect to a portion that is different from the model voice, the model voice is reproduced instead of the user voice as the enhancement process. As a result, an effect that the user can specifically specify a portion where there is a difference from the model voice and an effect that the correct pronunciation can be shown to the user regarding the portion where there is a difference are exhibited. Specifically, the difference highlighting process is performed as follows. The CPU 101 multiplies the model voice and the user voice whose lengths are equalized in step S103 by multiplying each by the volume coefficient. The volume coefficient is a parameter indicating the volume of the reproduced voice. For example, when the volume coefficient of the user voice is 1 and the volume coefficient of the model voice is 0, only the user voice is reproduced from the speaker 108. On the contrary, when the volume coefficient of the user voice is 0 and the volume coefficient of the model voice is 1, only the model voice is reproduced from the speaker 108.

  The CPU 101 refers to the data D and determines a volume coefficient for each frame. That is, in the user voice data, when the data D indicates that there is a difference, the CPU 101 sets the volume coefficient of the frame to 0. On the contrary, when the data D indicates that there is no difference from the model voice, the CPU 101 sets the volume coefficient of the frame to 1. On the other hand, in the exemplary audio data, when the data D indicates that there is a difference, the CPU 101 sets the volume coefficient of the frame to 1. Conversely, if the data D indicates that there is no difference from the model voice, the CPU 101 sets the volume coefficient of the frame to 0. The CPU 101 multiplies the user sound data and the model sound data by the volume coefficient obtained in this way, and mixes the user sound data and the model sound data. The CPU 101 outputs the mixed sound data obtained in this way to the sound processing unit 107.

  Subsequently, the audio processing unit 107 performs digital / analog conversion on the input mixed audio data and outputs the converted audio data to the speaker 108 as an audio signal. The enhanced sound is reproduced from the speaker 108 (step S107). By listening to this voice, the user can specifically identify a portion where there is a difference between the user's own voice and the model voice, and can know the correct pronunciation of the portion where there is the difference.

<Second Embodiment>
Subsequently, a second embodiment of the present invention will be described. In the following description, common reference numerals are assigned to elements common to the first embodiment, and description thereof is omitted.
FIG. 8 is a block diagram showing a functional configuration of the language learning device 2 according to the second embodiment of the present invention. If only a different part from the language learning apparatus 1 which concerns on 1st Embodiment is demonstrated, the difference point emphasis part 51 will be based on the difference point extracted in the difference point extraction part 40 with respect to model audio | voice data and user audio | voice data, Image data representing an image in which the model voice and the user voice are visualized and the difference between the two is emphasized is generated. The difference point display unit 61 performs image display based on the image data generated by the difference point emphasizing unit 51. Since the hardware configuration of the language learning device 2 is the same as the hardware configuration of the language learning device 1 shown in FIG.

  FIG. 9 is a flowchart showing the operation of the language learning device 2 according to the second embodiment of the present invention. Since the processing in steps S101 to S105 is the same as that in the first embodiment, the description thereof is omitted. When there is a part in the user voice that is different from the model voice (step S105: YES), the CPU 101 performs a difference point emphasis process (step S206). In the present embodiment, the difference between the model voice and the user voice is visually represented by an image.

  FIG. 10 is a diagram illustrating an image output in the present embodiment. The model audio image A and the user audio image B are displayed vertically arranged. Each audio image is mainly composed of two diagrams, a diagram (A-1, A-2) showing the volume and pitch, a diagram (B-1, B-2) showing the frequency characteristics, and the like. In these figures, the horizontal direction is the time axis. In step S206, the CPU 101 generates image data indicating an image as shown in FIG. 10 based on the model audio data and the user audio data. That is, the CPU 101 calculates the pitch (pitch) and volume for each frame from each audio data. As shown in FIG. 10, the CPU 101 determines the position (coordinates) in the vertical direction of the graphic in correspondence with the volume and the width of the graphic in correspondence with the pitch. The CPU 101 generates image data for displaying a graphic having the determined width at the determined position (the vertical coordinate) at the position on the time axis corresponding to the frame to be processed. In addition, the CPU 101 performs Fourier analysis on the speech waveform for each frame and performs frequency analysis. That is, a frequency (maximum amplitude frequency) having the maximum amplitude is obtained from the spectrum obtained by Fourier transform. The CPU 101 converts the maximum amplitude frequency into an identifier (color code) that specifies the color displayed on the display 105. For example, a table in which a frequency and a color code are recorded in association with each other is stored in advance in the HDD 104, for example, and the CPU 101 refers to this table and converts the maximum amplitude frequency into a color code. The CPU 101 generates image data for displaying the color specified by this color code at a position corresponding to the frame to be processed. Then, by changing the display color and brightness at the difference between the model voice and the user voice, the user can visually recognize the difference between the model voice and his / her voice.

  The CPU 101 outputs the generated image data to the display 105. The display 105 displays an image as shown in FIG. 10 according to the image data (step S207).

<Modification>
The present invention is not limited to the above-described embodiment, and various modifications can be made.
In each of the above-described embodiments, the function as the language learning device 1 or the language learning device 2 is realized by the CPU 101 executing the language learning program stored in the HDD 104. The functional configuration shown in FIG. 1 or FIG. The function as a language learning device may be realized in hardware by using an electronic circuit or the like corresponding to the element.

  Further, in each of the above-described embodiments, the mode in which the length of the user voice is matched with the model voice in the process of aligning the lengths of the model voice and the user voice (step S103) has been described. You may make it match with the length of.

In the first embodiment, the process for emphasizing the difference between the model voice and the user voice has been described by adjusting the volume. However, an additional process may be performed. For example, in the case where the part where the model voice and the user voice are different is a consonant, when reproducing the model voice, a specific frequency region may be amplified in order to hear the model voice more clearly. . Alternatively, for example, in general, it is known that the positions of the second formant and the third formant are different in pronunciation of l and r in English, and therefore, an emphasis process for amplifying a specific formant may be performed. This enhancement processing may be performed by the CPU 101 in accordance with a language learning program, or may be realized by hardware using a bandpass filter and an amplifier.
Also, in the aspect in which the difference is emphasized by adjusting the volume, the volume may be an intermediate volume value as well as the two levels “0” and “1”.

Moreover, although 2nd Embodiment demonstrated the aspect which displays a model audio | voice image and a user audio | voice image vertically, the display aspect of both images is not limited to this. For example, both images may be arranged side by side or displayed in an overlapping manner. In the case of overlapping display, it is desirable to change the display color between the model voice and the user voice.
Further, in the aspect in which the model voice image and the user voice image are displayed side by side, the example in FIG. 10 has described the aspect in which the image indicating the pitch and the volume and the image indicating the frequency characteristic are separately displayed. You may express by one image. For example, in the second embodiment, the image indicating the pitch and volume is displayed in a single color, but it may be displayed with a color indicating the frequency characteristic.
Further, the audio image is not limited to that illustrated in FIG. For example, the voice waveform may be displayed directly.
Further, the method of frequency analysis in step S206 is not limited to that described in the second embodiment. For example, the spectral envelope of the speech waveform may be obtained for each frame, and the formant frequency may be obtained. As the formant frequency, for example, any of the first to third formants can be used.

  In addition, it is possible to provide a language learning device that simultaneously has the function of performing the difference enhancement process described in the first embodiment and the image generation / display function described in the second embodiment.

It is a block diagram which shows the function structure of the language learning apparatus 1 which concerns on 1st Embodiment of this invention. 2 is a block diagram showing a hardware configuration of a language learning device 1. FIG. It is a figure which shows the content of model voice database DB1. It is a flowchart which shows operation | movement of the language learning apparatus 1 which concerns on the same embodiment. It is a figure which illustrates the waveform of model voice (A) and user voice (B). It is a flowchart which shows the process of step S103 in detail. It is a figure which illustrates the spectrum envelope of model voice (upper) and user voice (lower). It is a block diagram which shows the function structure of the language learning apparatus 2 which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the language learning apparatus 2 which concerns on the same embodiment. It is a figure which illustrates the image output in the embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Language learning apparatus, 2 ... Language learning apparatus, 10 ... Memory | storage part, 20 ... Input part, 30 ... Speech processing part, 40 ... Difference point extraction part, 50 ... Difference point emphasis part, 51 ... Difference point emphasis part, 60 DESCRIPTION OF SYMBOLS ... Audio | voice output part 61 ... Difference point display part 101 ... CPU, 102 ... RAM, 103 ... ROM, 104 ... HDD, 105 ... Display, 106 ... Microphone, 107 ... Voice processing part, 108 ... Speaker, 109 ... Keyboard, 110 ... Bus

Claims (4)

Storage means for storing the model voice;
An input means for inputting user voice;
Separating means for separating the exemplary voice and the user voice into a consonant part and a vowel part, respectively;
Processing means for changing the length of the vowel part of the exemplary voice or the user voice so that the lengths of the exemplary voice and the user voice are the same;
Comparison means for comparing the model voice and the user voice whose lengths are aligned by the processing means, and extracting a difference point ;
Difference highlighting means for highlighting the difference extracted by the comparison means;
Possess and output means for outputting sound differences is emphasized by the differences highlighted means,
The difference enhancement means increases the volume of the model voice for the part from which the difference is extracted by the comparison means, and increases the volume of the model voice for the part from which the difference is not extracted by the comparison means. Make the volume lower than the volume of the user voice
Language learning device characterized by that . Re-arrangement means for rearranging the consonant part of either one of the exemplary voice and the user voice so that the position on the time axis is the same as the other consonant part;
The processing means changes the length of the vowel part located between the consonant part rearranged by the rearrangement means and the consonant part appearing next to the consonant part on the time axis, respectively. The language learning device according to claim 1, wherein the exemplary voice and the user voice have the same length. Image generating means for extracting predetermined parameters for the user voice and the model voice, and generating image data indicating a figure corresponding to the amount of change of the parameters;
The language learning apparatus according to claim 1, further comprising: a display unit configured to display the image data generated by the image generation unit in a different display mode at the difference extracted by the comparison unit. 2. The language learning apparatus according to claim 1 , wherein the difference point emphasizing unit amplifies a specific frequency region in the model voice for a portion from which the difference point is extracted by the comparison unit.

Images