JP2010185967A - Pronunciation training device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pronunciation training device which allows a trainee to directly check the shape, movement and the like of lips for correct pronunciation and efficiently conduct training of pronunciation. <P>SOLUTION: The pronunciation training device 1 comprises: a language recognition means for recognizing a trainee's generating language generated by a trainee at a trainee's speech generating position and an instructor's generating language generated by an instructor at an instructor's speech generating position respectively; a conversion-into-character means for converting the trainee's generating language recognized by the language recognition means into visible characters visualizing the trainee's generating language and for converting the instructor's generating language recognized by the language recognition means into visible characters visualizing the instructor's generating language; and a display control means for performing such control that the characters visualizing the trainee's generating language are displayed at a position visualizing the trainee's speech in a pick-up image and the characters visualizing the instructor's generating language are displayed at a position visualizing the instructor's speech in a pick-up image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a pronunciation training device that displays characters in a display area, each of which visualizes a language uttered by a trainee and a language uttered by a trainer.

  Patent Document 1 discloses a voice display device that enables one person to easily perform correct utterance or pronunciation practice. In the voice display device of Patent Document 1, the display unit displays the detection pattern shape indicating the utterance or pronunciation of the speaker and the reference pattern shape indicating the correct utterance or pronunciation registered in advance, and the speaker displays the detection pattern shape. And the reference pattern shape can be viewed side by side on the screen of the display unit.

  Therefore, the speaker corrects his or her utterance by repeating the utterance or pronunciation while observing the detected pattern shape and the reference pattern shape displayed on the screen of the display unit so as to eliminate the difference between the two pattern shapes. Thus, the correct utterance or pronunciation can be acquired.

  Patent Document 2 discloses a sound source that estimates the direction of a sound source, collects an image near the estimated sound source position with a camera, and displays the sound source position on the image near the sound source position displayed on a display. Techniques relating to exploration systems are disclosed.

JP 2001-343890 A Japanese Patent Laid-Open No. 2002-181913

  By the way, in the voice display device described above, the trainer who is a speaker can visually recognize the reference pattern shape displayed on the screen, but the shape and movement of the lips when the voice corresponding to the reference pattern shape is emitted. Was not directly visible. For this reason, since the trainee cannot directly know the shape or movement of the lips, when performing training to eliminate the difference between the detection pattern and the reference pattern, the trainer emits a sound corresponding to the reference pattern shape. Assuming possible lip shapes and movements, it was necessary to repeat the process of bringing the detected pattern shape closer to the reference pattern shape through trial and error.

  Therefore, in the voice display device described above, it may take a long time for the trainee to bring the detected pattern shape closer to the reference pattern shape, and it is difficult to say that training of pronunciation is performed efficiently.

  The present invention is proposed in view of such a situation, and the trainer can directly confirm the shape and movement of the lips for correct pronunciation, and can efficiently train the pronunciation. An object of the present invention is to provide a pronunciation training device capable of performing

The pronunciation training device according to the first aspect of the present invention is a training that is a voice generated by the trainer and display means having a display area capable of displaying a captured image including a trainer image and a trainer image captured by a camera. A voice collecting means capable of collecting the voice of the leader and the voice of the leader, and the trainer based on the voice of the trainer and the voice of the leader collected by the sound collecting means. The position of the trainer voice generation position where the voice is uttered and the position of the trainer voice generation position where the voice of the instructor is uttered, the trainer voice generation position specified by the position specification means, and the display The trainer voice visualization display position in the captured image displayed in the area is correlated, and the instructor voice generation position specified by the position specifying means is displayed in the display area. Correlation means for correlating the display position of the trainer voice in the captured image, the trainer-generated language that is a language issued by the trainer at the trainer speech generation position, and the guidance at the trainer speech generation position Language recognition means for recognizing instructor-generated languages, which are languages issued by a trainee, and converting the trainer-generated language recognized by the language recognition means into visualized trainee-generated language visualized characters, and the language recognition means Character conversion means for converting the instructor-generated language recognized by the operator into visualized instructor-generated language visualized characters, and the training in the trainer speech visualization display position correlated with the trainer speech generation position by the correlating means In addition to performing control to display the person-generated language visualization characters, the correlation means correlates with the instructor voice generation position Characterized in that it comprises a display control means performs control to display the leader generation language visualization characters to the leader speech visualization position it is.
According to the pronunciation training device of the first aspect of the present invention, the trainer-generated language visualized character is displayed at the trainer speech visualization display position in the captured image including the trainer image and the trainer image by the display control means. While performing control, it performs control which displays a leader generation language visualization character in a leader voice visualization display position in the above-mentioned picked-up image.
Thereby, in the above display area, the captured image including the image of the trainer and the image of the instructor is displayed, and at the same time, the trainer-generated language visualization character and the instructor-generated language visualization character are displayed in the captured image. Can do.
Therefore, the trainee compares the trainer-generated language visualized character displayed in the display area with the trainer-generated language visualized character, and the trainer-generated language visualized character is different from the trainer-generated language visualized character. By confirming, it is possible to recognize that the language that the trainee himself has spoken is different from the language that the instructor has spoken.
In addition, the trainer recognizes that the language that the trainee himself has spoken is different from the language that the trainer has spoken, and at the same time the shape and movement of the lips represented by the trainer image displayed in the display area, etc. It is possible to match the shape and movement of the trainee's lips with the shape and movement of the instructor's lips while directly confirming.
Along with this, the trainer can efficiently perform training for correcting the language that the trainee himself speaks to be close to the language that the trainer speaks.

A second aspect of the present invention is directed to the first aspect, wherein the sound collecting means collects a pair of microphones arranged at a predetermined horizontal interval in order to collect the trainer voice and the instructor voice, respectively. In order to collect the microphone group arranged orthogonal to each other and the trainer voice and the instructor voice, the microphones constituting the pair of microphones are separated from each other by a facing interval that is the same as the horizontal interval. The position identification means uses a time difference when the trainee's voice reaches the pair of microphones and a time difference between the microphones and the opposite microphone, respectively. The trainer voice generation position is calculated by a hyperbolic method, and the instructor voice is converted into the pair of microphones. Using the time difference reaching the O emissions and the time difference to reach between the microphone and the counter microphone respectively, by hyperbola method, and calculates the leader sound generation position.
According to the invention of claim 2, the position specifying means includes a pair of microphones arranged at a predetermined interval constituting the microphone group, and a pair of microphones constituting the pair of microphones and a counter microphone. The trainer voice generation position and the teacher voice generation position are calculated by the hyperbolic method using the arrival time difference between the trainer voice and the leader voice to reach.
For this reason, even if the position where the trainer's voice is emitted and the position where the leader's voice is emitted change sequentially, the hyperbolic method can be used between each pair of microphones or each of the above microphones and the opposed microphones. The intersections of a plurality of hyperbolic trajectories that respectively pass between the two can be determined as the trainer voice generation position and the instructor voice generation position.

The invention of claim 3 is the trainer-generated language according to claim 1 or 2, wherein the trainer-generated language recognized by the language recognition means is compared with the trainer-generated language recognized by the language recognition means. Determination means for performing a correct answer determination that matches the instructor-generated language or an incorrect answer determination in which the trainer-generated language does not match the instructor-generated language, and a result of the correct answer determination by the determining means or an incorrect answer determination Determination result display control means for performing control to display the result in the display area.
According to the third aspect of the present invention, the determination result display control means performs control for displaying the result of the correct answer determination or the result of the incorrect answer determination in the display area.
Thereby, based on one of the determination results (correct answer determination result or incorrect answer determination result) displayed in the display area by the determination result display control means, the trainer is instructed by the language of the trainee himself / herself. It is possible to easily confirm at a glance whether or not the correct pronunciation corresponding to the language uttered by the person is made.

According to a fourth aspect of the present invention, in any one of the first to third aspects, the character conversion unit is configured to perform the training based on a volume of the trainer voice and a volume of the instructor voice collected by the sound collection unit. The size of the person-generated language visualized character and the size of the instructor-generated language visualized character are each changed, and the display control means is the size of the trainer-generated language visualized character changed by the character converting means, The trainer-generated language visualized character is displayed at the trainer-speech-visualized display position, and the guidance-speaker-spoken language-visualized character size is changed by the character conversion unit, and the guidance is displayed at the trainer-speech-visualized display position. It is characterized by displaying human-generated language visualization characters.
According to the fourth aspect of the present invention, the trainer-generated language visualized character is displayed at the trainer speech visualization display position with the size of the trainer-generated language visualized character changed by the display control means based on the volume of the trainee speech. In addition to displaying, the instructor-generated language visualized character is displayed at the instructor-speech-visualized display position with the size of the instructor-generated language visualized character changed based on the volume of the instructor voice.
For this reason, the trainee compares the size of the trainer-generated language visualized character and the size of the trainer-generated language visualized character to determine whether the volume of the trainer speech matches the volume of the trainer speech. can do.

A fifth aspect of the present invention includes the display position adjusting unit according to any one of the first to fourth aspects, wherein the trainer voice visualization display position and the instructor voice visualization display position are adjusted to arbitrary positions in the display area. It is characterized by that.
According to the invention of claim 5, the trainer voice visualization display position and the leader voice visualization display position can be adjusted to arbitrary positions in the display area by the display position adjusting means. The visualization display position and the instructor voice visualization display position can be adjusted to positions that do not block the trainee image and the instructor image in the display area.
Thus, the trainer separates the trainer-generated language visualization character at the trainer speech visualization display position and the trainer-generated language visualization character at the trainer speech visualization display position separately from the trainer image and the trainer image, respectively. Can be confirmed.
For this reason, it is possible to easily compare the trainee-generated language visualized character and the trainer-generated language visualized character, and compare the trainer image and the trainer image.

According to a sixth aspect of the present invention, in any one of the first to fifth aspects, the trainee is a hearing impaired person.
According to the invention of claim 6, in the display area, the captured image including the image of the hearing impaired person and the image of the instructor is displayed, and at the same time, the visually impaired language visualized characters and the instructor are displayed in the captured image. Emergence language visualization characters can be displayed.
Accordingly, the hearing impaired person compares the hearing-impaired person generated language visualized character displayed in the display area with the instructor-generated language visualized character, and the hearing impaired person generated language visualized character is By confirming that they are different, it is possible to recognize that the language spoken by the hearing impaired person is different from the language spoken by the instructor.
In addition, the hearing impaired person recognizes that the language spoken by the hearing impaired person is different from the language spoken by the instructor, and at the same time, the lip shape and the lip shape represented by the image of the instructor displayed in the display area. It is possible to match the shape and movement of the hearing impaired person's lips with the shape and movement of the instructor's lips while directly confirming the movement and the like.
Accordingly, the hearing impaired person can efficiently perform training for correcting the language emitted by the hearing impaired person so as to approach the language emitted by the instructor.

According to the pronunciation training device of the present invention, the display control means performs control to display the trainer-generated language visualization characters at the trainer speech visualization display position in the captured image including the trainer image and the trainer image. Then, control is performed to display the leader-generated language visualized character at the leader voice visualized display position in the captured image.
Thereby, in the above display area, the captured image including the image of the trainer and the image of the instructor is displayed, and at the same time, the trainer-generated language visualization character and the instructor-generated language visualization character are displayed in the captured image. Can do.
Therefore, the trainee compares the trainer-generated language visualized character displayed in the display area with the trainer-generated language visualized character, and the trainer-generated language visualized character is different from the trainer-generated language visualized character. By confirming, it is possible to recognize that the language that the trainee himself has spoken is different from the language that the instructor has spoken.
In addition, the trainer recognizes that the language that the trainee himself has spoken is different from the language that the trainer has spoken, and at the same time the shape and movement of the lips represented by the trainer image displayed in the display area, etc. It is possible to match the shape and movement of the trainee's lips with the shape and movement of the instructor's lips while directly confirming.
Along with this, the trainer can efficiently perform training for correcting the language that the trainee himself speaks to be close to the language that the trainer speaks.

It is a schematic block diagram of the pronunciation training apparatus of embodiment of this invention. It is a schematic block diagram of the personal computer which comprises the same pronunciation training apparatus. It is a flowchart regarding the process which the same pronunciation training apparatus performs. It is explanatory drawing of the sound source position specific process which the same pronunciation training apparatus performs. It is explanatory drawing of the display image coordinate transformation process which the same pronunciation training apparatus performs. It is a figure which shows the state by which the image was displayed on the display, when performing pronunciation training by the same pronunciation training apparatus. It is a figure which shows the state in which the image was displayed on the display, when the sound pressure level of a hearing impaired person's sound is low compared with the sound pressure level of a leader voice.

  Embodiment 1 of the present invention will be described with reference to FIGS. FIG. 1 is a schematic configuration diagram of a pronunciation training device 1 according to the present embodiment. The pronunciation training device 1 includes a measurement unit 10, an amplifier 20, a low-pass filter 30, an A / D converter 40, a personal computer 50, a video input / output unit 60, and a large display 70.

  As shown in FIG. 1, the measurement unit 10 includes support members 15A to 15C, a base 16, a CCD camera 17, a microphone support 18, and microphones M1 to M5. The base 16 is arrange | positioned at the upper part of support member 15A-15C. The microphone support base 18 is supported on the base 16 by a camera support member. A CCD camera 17 is fixed to the camera support member. Microphones M1 to M5 are attached to the microphone support base 18.

  In the measurement unit 10, the microphone M1 and the microphone M4 constitute a pair of microphones. The horizontal interval between the microphone M1 and the microphone M4 is kept at a predetermined distance. Further, the microphone M2 and the microphone M3 constitute another pair of microphones. The horizontal interval between the microphone M2 and the microphone M3 is maintained at a predetermined distance, similarly to the horizontal interval between the microphone M1 and the microphone M4. The pair of microphones M1 and M4 are arranged orthogonal to the other pair of microphones M2 and M3. A pair of microphones M1 and M4 and another pair of microphones M2 and M3 form a microphone group. The microphone M5 is disposed so as to protrude above the microphones M1 to M4. The facing distance between the microphone M5 and each of the microphones M1 to M4 is maintained at the same distance as the horizontal distance between the microphone M1 and the microphone M4 and the horizontal distance between the microphone M2 and the microphone M3. The microphone M5 is an example of the opposed microphone of the present invention, and the microphones M1 to M5 are examples of the sound collecting means of the present invention.

  Each microphone M <b> 1 to M <b> 5 is connected to the amplifier 20. The amplifier 20 amplifies the sound pressure signal transmitted from each of the microphones M1 to M5. The amplifier 20 is connected to the low pass filter 30. The low-pass filter 30 limits the frequency band that passes through the filter, and allows an audio signal of a predetermined frequency or lower (here, 4500 Hz or lower) to pass. The low pass filter 30 is connected to the A / D converter 40. The A / D converter 40 converts the sound pressure signal (analog signal) into a digital signal. The digital signal is transmitted to the personal computer 50.

  The CCD camera 17 is connected to the video input / output unit 60. The video input / output unit 60 converts the imaging signal (analog signal) transmitted from the CCD camera 17 into a digital signal. A digital signal (imaging signal) is transmitted to the personal computer 50 by the video input / output unit 60.

  The personal computer 50 is connected to the large display 70. Reference numeral 71 denotes a display area of the large display 70. The large display 70 is an example of the display means of the present invention.

  FIG. 2 is a schematic block diagram of the personal computer 50. The personal computer 50 includes a keyboard 51, an arithmetic processing unit 52, and a storage unit 53.

  The keyboard 51 is connected to the arithmetic processing unit 52. The keyboard 51 has the number of microphones, the distance between the microphone M1 and the microphone M4, the distance between the microphone M2 and the microphone M3, the distance between the microphone M5 and each of the microphones M1 to M4, and the frequency that passes through the low-pass filter 30. Used to input setting values.

  The arithmetic processing unit 52 is connected to the storage unit 53, the speaker 54, and the large display 70, respectively. The storage unit 53 includes a digital signal (sound pressure signal) arithmetic processing program storage unit 53A, a pronunciation training data processing program storage unit 53B, an image display control program storage unit 53C, and a data storage unit 53D. The digital signal (sound pressure signal) arithmetic processing program storage unit 53A stores a program for executing frequency analysis processing (S5), sound source position specifying processing (S6), display image coordinate conversion processing (S8), and the like, which will be described later. Yes. The pronunciation training data processing program storage unit 53B includes a pronunciation training language selection process (S3), a language recognition process (S7), a pronunciation training language display position adjustment process (S9), a character conversion process (S10), and a pronunciation. A program for executing the training correctness determination process (S11) is stored. The image display control program storage unit 53C stores a program for executing a pronunciation training image display process (S12) described later. The data storage unit 53D stores mel frequency cepstrum coefficient (MFCC) reference data obtained by collecting and analyzing a large amount of audio samples.

  As will be described later, the arithmetic processing unit 52 converts the image data of the pronunciation instructor and the hearing-impaired image captured by the CCD camera 17 and the sound produced by the instructor and the hearing-impaired person into characters. Based on the character image data thus generated, an image signal of the display image displayed in the display area 71, a signal related to the character image data, and a sound signal related to the sound produced by the instructor or the hearing impaired person are generated. Subsequently, the arithmetic processing unit 52 transmits the generated image signal or the like to the large display 70, and superimposes the captured image on the display area 71 to display the character image corresponding to the voice uttered by the instructor or the hearing impaired person, respectively. indicate. In addition, the arithmetic processing unit 52 outputs the above-described audio signal to the speaker 54, and the speaker 54 plays the language that the instructor or the hearing-impaired person respectively utters by voice.

  Next, a description will be given of a process in which the arithmetic processing unit 52 displays a captured image of each of the instructor and the hearing impaired person in the display area 71 and a character image obtained by converting voices emitted by the instructor or the hearing impaired person into characters. . A hearing impaired person is an example of a trainer of the present invention. When the pronunciation training device 1 is powered on, the arithmetic processing unit 52 executes an initial setting process (S1) as shown in FIG. In the initial setting process (S1), the number of microphones input by the keyboard 51 (here, five), the distance between the microphone M1 and the microphone M4, the distance between the microphone M2 and the microphone M3, the microphone M5, The process which memorize | stores the space | interval with microphone M1-M4 and the setting value of the frequency which passes the low-pass filter 30 in the data storage part 53D is performed.

  The arithmetic processing unit 52 executes an initial screen display process (S2) after the initial setting process (S1). In the initial screen display process (S2), by executing the program stored in the image display control program storage unit 53C, a language selection menu screen for speech training is displayed in the display area 71 as an initial image. Execute the process. The instructor can select any utterance training language from among the utterance training languages displayed on the selection menu screen. A training language display position adjustment button, which will be described later, is also displayed on the selection menu screen.

  The arithmetic processing unit 52 executes a pronunciation training language selection process (S3) after the initial screen display process (S2). In the pronunciation training language selection process (S3), it is determined whether or not the language for speech training has been selected by the instructor from the selection menu screen. Here, the arithmetic processing unit 52 determines whether or not the pronunciation training language data (pronunciation training language data) selected by the instructor is stored in the data storage unit 53D. In the pronunciation training language selection process (S3), it is repeatedly determined whether or not the pronunciation training language is selected until it is determined that the pronunciation training language is selected. In the pronunciation training language selection process (S3), character image data (for example, pronounce “hi”) is generated according to the pronunciation training language data selected by the instructor, and the character image data is generated. Processing to be stored in the data storage unit 53D is executed.

  The arithmetic processing unit 52 executes an input signal acquisition process (S4) after the pronunciation training language selection process (S3). In the input signal acquisition process (S4), the sound pressure level (sound pressure signal) and the voice of the leader (voice of the leader) and the voice of the hearing impaired (voice of the hearing impaired) For example, a process of acquiring an audio signal related to “hi” in FIG. 6, an audio signal related to the hearing impaired person's voice (for example, “i” in FIG. 6), and the above imaging signal is executed. Here, the sound pressure signal, the sound signal, and the imaging signal detected by the microphones M1 to M5 are input to the arithmetic processing unit 52 as digital signals as shown in FIG. Thereafter, the arithmetic processing unit 52 executes a process of storing the sound pressure signal (sound pressure level data), the sound signal (sound data), and the imaging signal (imaging data) in the data storage unit 53D.

  The arithmetic processing unit 52 executes a frequency analysis process (S5) after the input signal acquisition process (S4). In the frequency analysis processing (S5), the sound pressure acquired by the input signal acquisition processing (S4) using the program (Fourier transform processing program) stored in the digital signal (sound pressure signal) arithmetic processing program storage unit 53A. A process of analyzing the sound pressure level of the signal and analyzing the spectrum distribution of the frequency is executed. Thereafter, in the frequency analysis process (S5), the MFCC is calculated based on the spectrum distribution of the frequency. Here, as an example, twelve MFCCs were calculated based on each frequency spectrum distribution of the instructor voice and the hearing impaired voice by frequency analysis processing (S5). In addition, in the frequency analysis process (S5), a process of storing MFCC data in the data storage unit 53D is executed.

  The arithmetic processing unit 52 executes the sound source position specifying process (S6) after the frequency analysis process (S5). In the sound source position specifying process (S6), a microphone group origin position O (see FIG. 4) and positions P1, P2 are obtained by a hyperbolic method using a program stored in the digital signal (sound pressure signal) arithmetic processing program storage unit 53A. A process of calculating the horizontal angle θ (see FIG. 4) between (see FIG. 4) is executed. The data of the horizontal angle θ is stored in the data storage unit 53D. The microphone group origin position O is a position where a straight line connecting the microphone M1 and the microphone M4 intersects with a straight line connecting the microphone M2 and the microphone M3. The position P1 is a position where the leader voice is emitted (leader voice generation position), and the position P2 is a position where the hearing impaired person voice is emitted (deaf person voice generation position). In the drawing, a symbol L1 is a horizontal distance from the position O1 facing the microphone group origin position O to the positions P1 and P2, and a symbol L2 is a distance from the microphone group origin position O to the position O1. A symbol L4 is a distance from the position O1 where an intersection O2 between a vertical line extending downward from the position O1 and a horizontal line extending leftward from the positions P1 and P2 is separated.

The value of the horizontal angle θ is the distance between the microphone M1 and the microphone M4, the distance between the microphone M2 and the microphone M3, the voice of the instructor at the position P1 or the voice of the hearing impaired person at the position P2, and the pair of microphones M1 and M4. , And the voice of the instructor at position P1 or the voice of the hearing impaired person at position P2 varies depending on the time difference of reaching the other pair of microphones M2 and M3. The horizontal angle θ is calculated using the following formula (1). Note that DX is the arrival time difference between the leader voice or hearing impaired person voice in the pair of microphones M1 and M4, and DY is the arrival time difference between the leader voice or hearing impaired person voice in the other pair of microphones M2 and M3. is there.
θ = tan ⁻¹ (DY / DX) (1)

Further, in the sound source position specifying process (S6), by using the above-described microphone M5 in addition to the microphones M1 to M4, the following angle (2) is used to look up the microphone group origin position O from the positions P1 and P2. φ (see FIG. 4) is calculated. The data of the elevation angle φ is stored in the data storage unit 53D. The following DZ1 is the arrival time difference between the leader voice or the hearing impaired voice in the microphones M5 and M1, DZ2 is the arrival time difference between the leader voice or the hearing impaired voice in the microphones M5 and M2, and DZ3 is the microphone M5, The arrival time difference between the leader voice or the hearing impaired voice in M3 and DZ4 indicate the arrival time difference between the leader voice or the hearing impaired voice in the microphones M5 and M4, respectively.
φ = tan ⁻¹ {(DZ1 + DZ2 + DZ3 + DZ4) / [2 × √3 × (√DX ² + √DY ² )]} (2)

  In the sound source position specifying process (S6), by using the hyperbolic method, a hyperbolic trajectory passing between the microphone M1 and the microphone M4, a hyperbolic trajectory passing between the microphone M2 and the microphone M3, the microphone M5 and each microphone M1. The points at which the hyperbola trajectories passing through M4 intersect can be calculated as positions P1 and P2. The arithmetic processing unit 52 is an example of a position specifying unit of the present invention.

  The arithmetic processing unit 52 executes a language recognition process (S7) after the sound source position specifying process (S6). In the language recognition process (S7), the language recognition program is read from the pronunciation training data processing program storage unit 53B. The language recognition program is a program that extracts words using a Japanese recognition function. Thereafter, in the language recognition process (S7), the voice of the instructor ("hi" in FIG. 6) and the hearing impaired person voice ("I" in FIG. 6) acquired in the input signal acquisition process (S4) by the language recognition program. )) Is extracted from each voice signal (here, “hi” and “i”). The arithmetic processing unit 52 is an example of the language recognition means of the present invention. Japanese “hi” is an example of the instructor utterance language of the present invention, and Japanese “i” is an example of the trainer utterance language of the present invention.

  The arithmetic processing unit 52 executes display image coordinate conversion processing (S8) after language recognition processing (S7). In the display image coordinate conversion process (S8), a process for correlating the position P1 (leader voice generation position) and the position P2 (deaf person voice generation position) with the display position of the display area 71 is executed. Here, as shown in FIG. 5, the horizontal direction X of the display area 71 is associated with the horizontal angle θ and the vertical direction of the display area 71 is associated with the elevation angle φ, and the display area 71 corresponding to the positions P1 and P2. The display position of is calculated.

  Thereafter, the arithmetic processing unit 52 calculates a corrected horizontal angle obtained by adding an arbitrary horizontal angle to the horizontal angle θ calculated by the sound source position specifying process (S6), and the elevation angle φ calculated by the sound source position specifying process (S6). A process of calculating a corrected elevation angle obtained by adding an arbitrary elevation angle φ to is executed. As a result, the display position P5 (see FIG. 6) of the display area 71 corresponding to the corrected horizontal angle and the display position P6 (see FIG. 6) of the display area 71 corresponding to the corrected elevation angle are indicated to the instructor 75 (see FIG. 6). .) Of the mouth and the image of the mouth of the hearing impaired person 76 (see FIG. 6) can be adjusted to positions where they are not obstructed.

  Subsequently, in the display image coordinate conversion process (S8), a process of storing the data of the display positions P5 and P6 in the data storage unit 53D is executed. The arithmetic processing unit 52 is an example of the correlation means of the present invention. The display position P5 is an example of the instructor voice visualization display position of the present invention, and the display position P6 is an example of the trainer voice visualization display position of the present invention.

  The arithmetic processing unit 52 executes a pronunciation training language display position adjustment process (S9) after the display image coordinate conversion process (S8). In the pronunciation training language display position adjustment process (S9), it is determined whether or not the adjustment data (horizontal angle θ and elevation angle φ) of the display positions P5 and P6 is stored in the data storage unit 53D. Each adjustment data can be set to an arbitrary value by the instructor or the hearing impaired person by specifying the horizontal angle θ and the elevation angle φ using the training language display position adjustment button described above. Thereby, for example, as shown in FIG. 6, the display positions P5 and P6 corresponding to the arbitrarily set values are adjusted to positions that do not block the image of the instructor 75 and the image of the hearing impaired person 76, respectively. Can do. The arithmetic processing unit 52 and the training language display position adjustment button are examples of the display position adjustment means of the present invention.

  Thereafter, in the pronunciation training language display position adjustment process (S9), when it is determined that each adjustment data is stored in the data storage unit 53D, the data storage unit 53D is subjected to the display image coordinate conversion process (S8). The process of overwriting each adjustment data on the data of the display positions P5 and P6 stored in the above is executed.

  On the other hand, in the pronunciation training language display position adjustment process (S9), if it is determined that each adjustment data is not stored in the data storage unit 53D, the display image coordinate conversion process (S8) stores it in the data storage unit 53D. A process of holding the data at the displayed positions P5 and P6 is executed.

  The arithmetic processing unit 52 executes a character conversion process (S10) after the pronunciation training language display position adjustment process (S9). In the character conversion process (S10), the character conversion program is read from the pronunciation training data processing program storage unit 53B. The character conversion program is a program for converting words extracted by the language recognition process (S7) into characters. Here, the words (“hi” and “i”) extracted by the language recognition process (S7) are converted into the characters “hi” and “i”. As shown in FIG. 6 to be described later, the characters “Hi” and “I” are visualized by being displayed in the display area 71. Accordingly, the instructor 75 and the hearing impaired person 76 can visually confirm the characters “Hi” and “I”. The image data of each character “HI” and “I” is stored in the data storage unit 53D. The arithmetic processing unit 52 is an example of character conversion means of the present invention. The character “HI” is an example of the instructor utterance language visualization character of the present invention, and the character “I” is an example of the trainer utterance language visualization character of the present invention.

  Further, in the character conversion process (S10), the sound pressure level data acquired by the input signal acquisition process (S4) is read from the data storage unit 53D. Next, in the character conversion process (S10), the sound pressure level data is compared with data of a preset threshold value. Here, according to the result of comparison between the sound pressure level and the threshold data, character image data is generated in which the size of each character of “Hi” and “I” is set to large, medium, and small. . Character image data set to large, medium, and small sizes is stored in the data storage unit 53D. The sound pressure level of the hearing impaired person's voice is an example of the volume of the hearing impaired person's voice of the present invention, and the sound pressure level of the leader's voice is an example of the volume of the leader's voice of the present invention.

  The arithmetic processing unit 52 executes a pronunciation training correctness determination process (S11) after the character conversion process (S10). In the pronunciation training correctness determination process (S11), the MFCC reference data and the MFCC data calculated by the frequency analysis process (S5) are read from the data storage unit 53D. Thereafter, in the pronunciation training correctness determination process (S11), a reference distribution range of the MFCC is determined based on the reference data. Further, in the pronunciation training correctness determination process (S11), the distribution range of the MFCC of the instructor voice is determined based on the 12 MFCCs described above, and other 12 MFCCs different from the 12 MFCCs are determined. Based on this, the distribution range of the MFCC of the hearing impaired voice is determined. Subsequently, in the pronunciation training correctness determination process (S11), the distribution range of the MFCC of the leader voice and the distribution range of the MFCC of the hearing impaired voice are respectively compared with the reference distribution range of the MFCC, and the distribution of the MFCC of the leader voice. A guidance voice reference distribution range including the range and a hearing impaired voice reference distribution range including the MFCC distribution range of the hearing impaired voice are respectively selected. Next, in the pronunciation training correctness determination process (S11), it is determined whether or not the hearing impaired person's voice reference distribution range is included in the leader voice reference distribution range.

  In the pronunciation training correct / incorrect determination process (S11), when it is determined that the voice reference distribution range of the hearing impaired person is included in the leader voice reference distribution range, the voice of the leader (for example, “hi”) and the voice of the hearing impaired person (“ Hi)) matches, and the correct answer judgment that the hearing impaired person is speaking correctly in accordance with the instructor is performed. Further, in the pronunciation training correct / incorrect determination process (S11), character image data (for example, “correctly pronounced”) for displaying the determination result is generated according to the correct determination result, and the character image data is stored in the data. Store in the unit 53D.

  On the other hand, when it is determined that the voice reference distribution range of the hearing impaired person is not included in the leader voice reference distribution range, the voice of the leader (eg, “hi”) matches the voice of the hearing impaired person (eg, “i”). Without correct judgment, the hearing-impaired person is making a wrong utterance different from the leader. In the pronunciation training correct / incorrect determination process (S11), character image data for displaying the determination result (for example, “pronounced pronunciation in error”) is generated according to the result of the incorrect answer determination, and the character image data is stored as data. Store in the unit 53D. The arithmetic processing unit 52 is an example of a determination unit of the present invention.

  The arithmetic processing unit 52 executes a pronunciation training image display process (S12) after the pronunciation training correctness determination process (S11). In the pronunciation training image display process (S12), using the program stored in the image display control program 53C, the image data of the captured image corresponding to the imaging signal acquired by the input signal acquisition process (S4), the character conversion process (S10) Based on the character image data and the like generated by), the voice of the instructor or the hearing impaired person is sent to the display position determined by the display image coordinate conversion process (S8) and the pronunciation training language display position adjustment process (S9). A process of displaying an image converted into characters is executed. The arithmetic processing unit 52 is an example of the display control means of the present invention.

  FIG. 6 shows a case where the leader 75 selects “hi” as a pronunciation training word on the selection menu screen described above, and the hearing impaired person 76 makes a wrong utterance different from the leader 75. An example of the display image is shown. At the position P1 (see FIG. 4), the instructor 75 utters “hi” and the hearing impaired person 76 utters “hi” even if he / she intends to utter “hi”. In this case, in the pronunciation training image display process (S12), the following process is executed. In the pronunciation training image display process (S12), the character image data (pronounced “hi”) stored in the data storage unit 53D by the pronunciation training language selection process (S3) from the data storage unit 53D, Character image data (character image data of “hi” and “i”) stored in the data storage unit 53D by the character conversion process (S10), and stored in the data storage unit 53D by the pronunciation training correctness determination process (S11) Character image data (“pronounced pronunciation”), etc. Further, in the pronunciation training image display process (S12), the captured image data and the like stored in the data storage unit 53D are read out by the input signal acquisition process (S4).

  As shown in FIG. 6, in the pronunciation training image display process (S12), a character image (pronounce “hi”) is displayed over the captured image (the instructor 75 and the hearing impaired person 76). Thereafter, based on the data stored in the data storage unit 53D by the display image coordinate conversion process (S8) and the pronunciation training language display position adjustment process (S9), the display position P5 is within the area surrounded by a circle. While displaying the character image of “Hi”, the character image of “I” is displayed at the display position P6 in the area surrounded by a circle. In this example, the sound pressure level of the hearing impaired person's voice matches the sound pressure level of the instructor's voice, and the size of the character image of “I” is the same as the size of the character image of “Hi” showed that.

  Subsequently, as shown in the figure, the display area 71 displays a character image “pronounced pronunciation”. Thereby, it is displayed to the hearing impaired person 76 that the voice of the hearing impaired person 76 is different from the voice of the instructor 75 and that the character images at the display positions P5 and P6 are different from each other. Each can be notified through the image of the area 71. The hearing impaired person 76 directly confirms the shape and movement of the lips of the instructor 75 while looking at the image of the display area 71, and also displays the character image (determined result) of the display positions P5 and P6. While observing "pronounced pronunciation" etc.), the language that the hearing impaired person 76 speaks is brought close to the language that the instructor 75 speaks (here "hi"). The arithmetic processing unit 52 is an example of a determination result display control unit of the present invention.

  On the other hand, when the language of the hearing impaired person 76 coincides with the language of the instructor 75 (“hi”) and the correct answer determination is made by the pronunciation training correct / incorrect determination process (S11), the display position P6 includes A character image of “Hi” is displayed, and a character image of “Properly pronounced” is displayed in place of the character image of “May be pronounced incorrectly” in FIG. Thereby, it is possible to inform the hearing impaired person 76 that the language emitted by the hearing impaired person 76 matches the language emitted by the instructor 75.

  When the sound pressure level of the hearing impaired person voice does not match the sound pressure level of the instructor voice, the character image data stored in the data storage unit 53D (character image data of “hi” and “i”) is stored. Based on this, as shown in FIG. 7, the character image “HI” at the display position P5 and the character image “I” at the display position P6 are displayed in different sizes. The illustrated example shows that the character image of “I” is displayed smaller than the character image of “Hi” because the sound pressure level of the hearing impaired person's voice is lower than the sound pressure level of the instructor's voice.

  After the pronunciation training image display process (S12), the arithmetic processing unit 52 determines whether or not to continue the pronunciation training (S13). Here, the arithmetic processing unit 52 determines that the power of the pronunciation training device 1 is on or off. In S13, when it is determined that the power is on and the pronunciation training is continued, the above-described processes (S2 to S12) are repeated. On the other hand, if it is determined in S13 that the power is off and the pronunciation training is to be ended, the above-described processes (S2 to S12) are ended.

<Effect of this embodiment>
In the pronunciation training device 1 of the present embodiment, each image of the hearing impaired person 76 and the instructor 75 is displayed in the display area 71 by the pronunciation training image display process (S12), and at the same time at the display position P6 in the display area 71. The character image “I” and the character image “HI” can be displayed at the display position P5 in the display area 71, respectively.
Therefore, the hearing impaired person 76 compares the character image of “hi” displayed in the display area 71 with the character image of “i”, and the hearing impaired person sound (“i”) is instructed by the instructor voice (“hi”). It is possible to recognize that the language uttered by the hearing impaired person 76 is different from the language uttered by the instructor 75.
In addition, the hearing impaired person 76 is recognized by the image of the leader 75 displayed in the display area 71 while recognizing that the language emitted by the hearing impaired person 76 is different from the language emitted by the leader 75. It is possible to match the shape and movement of the hearing impaired person 76 with the shape and movement of the instructor 75 while directly confirming the shape and movement of the lip.
Accordingly, the hearing impaired person 76 can efficiently perform training for correcting the language emitted by the hearing impaired person 76 so as to be close to the language emitted by the instructor 75.

In addition, by the sound source position specifying process (S6), the time difference between the voice of the instructor at position P1 and the voice of the hearing impaired person at position P2 reaching the pair of microphones M1 and M4, respectively, The positions P1, P2 are determined by the hyperbolic method using the time difference between the microphones M2 and M3 and the time difference between the microphone M5 and each of the microphones M1 to M4. Is calculated.
For this reason, even if the position P1 where the instructor's voice is emitted and the position where the hearing impaired person's voice P2 is sequentially changed, a hyperbolic locus passing between the microphone M1 and the microphone M4 by the hyperbolic method. The hyperbolic trajectory passing between the microphone M2 and the microphone M3 and the hyperbolic trajectory passing between the microphone M5 and each of the microphones M1 to M4 can be sequentially calculated.
Accordingly, a hyperbolic trajectory passing between the microphone M1 and the microphone M4, a hyperbolic trajectory passing between the microphone M2 and the microphone M3, and a hyperbolic trajectory passing between the microphone M5 and each of the microphones M1 to M4, respectively. The intersecting points can be calculated as the position P1 where the instructor voice is emitted and the position P2 where the hearing impaired person voice is emitted.

Further, by the pronunciation training image display process (S12), the character image for displaying the correct answer determination result (“pronounced correctly”) or the character image for the incorrect answer determination result (“pronounced pronounced incorrectly”) is displayed in the display area 71. ")") Is displayed.
Thus, based on the character image (“pronounced pronounced correctly” or “pronounced pronounced”) displayed in the display area 71, the hearing impaired person 76 emitted the hearing impaired person 76. It is possible to easily confirm at a glance whether or not the correct pronunciation of the language corresponding to the language issued by the instructor 75 is made.

In addition, when the sound pressure level of the hearing impaired person's voice matches the sound pressure level of the instructor's voice, the pronunciation training image display process (S12) performs display at the display position P6 as shown in FIG. The size of the character image “I” of the hearing impaired person voice is the same as the size of the character image “HI” of the instructor voice at the display position P5.
On the other hand, when the sound pressure level of the hearing impaired person voice is lower than the sound pressure level of the instructor voice, the pronunciation training image display process (S12) causes the hearing impaired person at the display position P6 as shown in FIG. The size of the voice character image “I” is smaller than the size of the character image “HI” of the instructor voice at the display position P5.
For this reason, the hearing impaired person 76 compares the size of the character image of “I” with the size of the character image of “Hi”, and the sound pressure level of the hearing impaired person voice matches the sound pressure level of the instructor voice. It can be determined whether or not.

In addition, as described above, by the pronunciation training language display position adjustment process (S9), as shown in FIGS. 6 and 7, the display positions P5 and P6 are changed to the images of the instructor 75 and the hearing impaired person 76. Each image can be adjusted to a position that does not block the image.
As a result, the hearing impaired person 76 converts the character image “hi” of the leader voice at the display position P5 and the character image “i” of the hearing person voice at the display position P6 to the image of the hearing impaired person 76 and the leader 75. Each can be confirmed separately from the image.
For this reason, it is possible to easily compare the character image “I” of the hearing impaired person voice with the character image “hi” of the instructor voice, and the image of the hearing impaired person 76 and the image of the instructor 75, respectively. it can.

  The present invention is not limited to the embodiment described above, and can be implemented by appropriately changing a part of the configuration without departing from the spirit of the invention. The pronunciation training apparatus according to the present embodiment displays not only one hearing impaired person 76 but also each captured image of the two or more hearing impaired persons 76 and the instructor 75 in the display area 71 and two or more hearing impaired persons. The person 76 may apply the pronunciation training while directly confirming the shape and movement of the lips of the instructor 75 by looking at each captured image. Thereby, two or more hearing impaired persons 76 can perform pronunciation training at a time, and two or more hearing impaired persons 76 can perform pronunciation training efficiently.

  In addition, the pronunciation training device of the present embodiment has three levels of large, medium, and small characters at the display positions P5 and P6 based on the sound pressure level data acquired by the input signal acquisition process (S4). Not limited to the setting, each display position P5, P6 may display characters such as “hi” and “i” depending on the size set in a plurality of stages of three or more. .

  Furthermore, the pronunciation training device according to the present embodiment uses a method for determining whether the voice of the instructor and the hearing impaired person match, and is not limited to using MFCC, but is acquired by the input signal acquisition process (S4). A speech feature vector calculated based on the speech signal, a combination of the speech feature vector and MFCC, or a probabilistic model (eg, Hidden Markov Model) may be used.

  In addition, the pronunciation training device of the present embodiment is not limited to the pronunciation training for the hearing impaired person 76, and may be applied to pronunciation training performed by, for example, an infant or a foreigner who learns Japanese.

  In addition, the pronunciation training device of the present embodiment is not limited to word pronunciation training, and may be applied to pronunciation training of simple sentences and long sentences. Furthermore, the pronunciation training device is not limited to Japanese pronunciation training, and may be applied to foreign language pronunciation training.

  In addition to determining whether or not the hearing impaired person 76 is making a correct utterance in accordance with the instructor 75, the pronunciation training device of the present embodiment is suitably used to correct the pronunciation of the hearing impaired person 76. May be displayed in the display area 71. For example, a message image that informs the hearing impaired person 76 specifically of the shape and movement of the lips or the like (for example, “Round the lips and pronounce them”) may be displayed in the display area 71. Thus, the hearing impaired person 76 repeats pronunciation training in accordance with the message image displayed in the display area 71, thereby changing the shape and movement of the lip of the hearing impaired person 76 to the shape and movement of the lip of the instructor 75. Can be approached. Therefore, the pronunciation of the hearing impaired person 76 can be corrected to match the pronunciation of the instructor 75.

  1 .... Pronunciation training device, 17 .... CCD camera, 52 ... Operation processing unit, 70 ... Large display, 71 ... Display area, 75 ... Leader, 76 ... Hearing impaired, M1-M5 ... Microphone, P1 .. Leader voice generation position, P2 .. Hearing impaired voice generation position, P5, P6 .. Display area display position

Claims (6)

Display means having a display area capable of displaying a captured image including an image of a trainee and an image of an instructor captured by a camera;
Sound collecting means capable of collecting each of the trainer voice that is the voice that the trainer has uttered and the leader voice that is the voice that the trainer has uttered;
Based on the trainer voice and the instructor voice collected by the sound collection means, the trainer voice generation position from which the trainer voice is generated and the instructor voice generation position from which the instructor voice is generated, respectively. A position identification means to identify;
The trainer voice generation position specified by the position specifying means and the trainer voice visualization display position in the captured image displayed in the display area are correlated with each other, and the instructor voice generation specified by the position specifying means is correlated. Correlation means for correlating the position and the leader voice visualization display position in the captured image displayed in the display area;
Language recognition means for recognizing a trainer-generated language that is a language issued by the trainer at the trainer speech generation position and a leader-generated language that is a language issued by the trainer at the leader speech generation position;
The trainer-generated language visualized characters visualized by the trainer-generated language recognized by the language recognizing means and the trainer-generated language recognized by the language recognizer are converted to visualized teacher-generated language visualized characters. Character conversion means;
Control is performed to display the trainer-generated language visualization character at the trainer-speech visualization display position correlated with the trainer-speech generation position by the correlation unit, and the correlation unit correlates with the teacher-speech generation position. Display control means for performing control to display the leader-generated language visualization characters at the attached leader voice visualization display position;
A pronunciation training device comprising: The sound collecting means includes
A microphone group in which a pair of microphones arranged at predetermined horizontal intervals are orthogonal to each other in order to collect the trainee voice and the instructor voice, respectively,
In order to collect the trainer's voice and the instructor's voice, respectively, an opposing microphone disposed at an opposing interval that is the same interval as the horizontal interval from each microphone constituting the pair of microphones. ,
The position specifying means includes
The trainer voice uses the time difference to reach the pair of microphones and the time difference to reach between each microphone and the opposing microphone, respectively, to calculate the trainer voice generation position by the hyperbolic method,
The instructor voice generation position is calculated by the hyperbolic method using the time difference when the instructor voice reaches the pair of microphones and the time difference between the microphones and the opposing microphone, respectively. The pronunciation training device according to claim 1. Comparing the trainer-generated language recognized by the language recognizing means with the teacher-generated language recognized by the language recognizing means, the correct answer determination or training that the trainer-generated language matches the leader-generated language A determination means for performing an incorrect answer determination in which a person-generated language does not match the leader-generated language;
Determination result display control means for performing control to display the result of the correct answer determination or the result of the incorrect answer determination by the determining means in the display area;
The pronunciation training apparatus according to claim 1, comprising: The character conversion means, based on the volume of the trainer voice and the volume of the instructor voice collected by the sound collection means, the size of the trainer-generated language visualized characters and the instructor-generated language visualized characters Change the size of each,
The display control means includes
In the size of the trainer-generated language visualized character changed by the character conversion means, the trainer-generated language visualized character is displayed at the trainer voice visualized display position,
4. The instructor-generated language visualized character is displayed at the instructor-speech-visualized display position with the size of the instructor-generated language visualized character changed by the character conversion means. The pronunciation training device described in Crab.   The pronunciation according to any one of claims 1 to 4, further comprising display position adjusting means for adjusting the trainer voice visualization display position and the instructor voice visualization display position to an arbitrary position in the display area. Training device.   The pronunciation training apparatus according to claim 1, wherein the trainee is a hearing impaired person.

Images