JP2016156943A - Display controller, display control method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display controller, a display control method and a program that enable a speaker to grasp, in an easier-to-understand manner, a difference in way of speaking between a voice as a model and a speaker's voice based upon a pitch, sound pressure and a pronunciation feature quantity.SOLUTION: A voice information display device is configured: to determine a position, a width and a color of a model voice line representing change of a voice as a model in reading a character string aloud based pon a pitch, sound pressure and a pronunciation feature quantity specified from model voice waveform data; to determine a position, a width and a color of a speaker voice line showing change of a voice that a speaker speaks in reading the character string aloud based pon a pitch, sound pressure and a pronunciation feature quantity specified from speaker voice waveform data; to display the model voice line and speaker voice line on a screen so that they can be compared with each other; and to display the character string on the screen along a time-base direction.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technical field such as a system capable of visually expressing a voice uttered when a speaker reads a character string aloud.

  In recent years, for the purpose of supporting language learning, utterance utterance training, and the like, there is known a technique for visually expressing speech uttered when a speaker reads a character string aloud. For example, Patent Document 1 discloses a system that displays a graphic corresponding to the reference voice and representing the timing of utterance, the length of utterance, the pitch, and the prompt sound, and changing the color of the utterance part. . On the other hand, Patent Literature 2 discloses a system that can arbitrarily set data of a singing method as a model and visually grasp a difference in a practitioner's singing method with respect to the singing method. In this system, the transition of the pitch of the model song is displayed as a broken line according to the pitch data, and the color of the area between the broken line and the horizontal axis is displayed so as to change according to the value of the volume data of the model song. ing.

JP 2003-186379 A JP 2006-276893 A

  However, although there has been a technology to display lines and graphs using parameters such as pitch and volume as parameters, the speaker can understand the difference in how the voice is used as an example and the voice of the speaker at a glance. A system that can be easily grasped was not known.

  The present invention has been made in view of the above points, and the difference in the manner of utterance between the voice that serves as a model and the voice of the speaker based on the pitch, the sound pressure, and the pronunciation feature amount is described. The present invention provides a display control device, a display control method, and a program that can be understood at a glance.

  In order to solve the above-mentioned problem, the invention according to claim 1 is a first example showing text data of a character string composed of a plurality of characters and a waveform of a voice as a model when the character string is read aloud. Storage means for storing a pitch, a sound pressure, and a pronunciation feature amount specified for each predetermined unit based on the speech waveform data, and a second waveform indicating a speech waveform generated when the speaker reads the character string aloud. Input means for inputting speech waveform data, specification means for specifying a pitch, sound pressure, and pronunciation feature amount for each predetermined unit based on the first speech waveform data, and a pitch stored in the storage means First control means for displaying on the screen a first line that extends in the time axis direction and changes in time series in an axis direction orthogonal to the time axis direction, based on the sound pressure and the sound production feature amount And the character string based on the text data Based on the second control means for displaying each character constituting the screen on the screen along the time axis direction, the pitch, the sound pressure, and the pronunciation feature quantity specified by the specifying means, the character extends in the time axis direction. And a third control means for displaying a second line that changes in time series in an axial direction orthogonal to the time axis direction on the screen so as to be comparable to the first line. The first control unit and the third control unit each determine a position of the line in the axial direction orthogonal to the time axis direction for each predetermined unit based on the pitch, and the sound A second determination unit that determines the width of the line for each predetermined unit based on pressure, and a third determination unit that determines the color of the line for each predetermined unit based on the pronunciation feature amount. It is characterized by providing.

  According to a second aspect of the present invention, in the display control device according to the first aspect, the second control means determines the position of the character in the axial direction orthogonal to the time axis direction based on the pitch. A fourth determination unit that determines each predetermined unit, and a fifth determination unit that determines the size of the character for each predetermined unit based on the sound pressure.

  According to a third aspect of the present invention, in the display control device according to the first or second aspect, the pronunciation feature amount is a feature amount that can be distinguished by a spectrum of the speech waveform data, and the third determination unit includes: The color of the line is determined according to the pronunciation feature amount.

  According to a fourth aspect of the present invention, in the display control device according to the third aspect, the predetermined unit is a character unit, and the storage means is specified for each character unit based on the first speech waveform data. The vowel is stored as the generated pronunciation feature value, the specifying means specifies the vowel as the pronunciation feature value for each character unit based on the second speech waveform data, and the third determining unit According to another aspect of the present invention, a color corresponding to each of the generated vowels is determined as the color of the line.

  According to a fifth aspect of the present invention, in the display control device according to the fourth aspect, the storage means uses the vowel and the vowel as the pronunciation feature amount specified for each character based on the first speech waveform data. Sound components other than vowels are stored, and the specifying means specifies sound components other than vowels and vowels as the pronunciation feature amount for each character unit based on the second speech waveform data, and the third determining unit Is characterized in that a mixed color of a color corresponding to the specified vowel and a color corresponding to a sound component other than the vowel is determined as the color of the line.

  According to a sixth aspect of the present invention, in the display control apparatus according to the third aspect, the predetermined unit is a predetermined time unit, and the storage unit is configured to perform predetermined time unit based on the first speech waveform data. At least a first formant frequency and a second formant frequency are stored as the pronunciation feature quantity specified in step (b), and the specifying means at least as the pronunciation feature quantity for each predetermined time unit based on the second speech waveform data. A first formant frequency and a second formant frequency are specified, and the third determining unit selects a color corresponding to a combination of the specified first formant frequency value and the specified second formant frequency value. It is determined as the color of the line.

  According to a seventh aspect of the present invention, in the display control device according to the sixth aspect, a predetermined reference color is set for a combination of a predetermined first formant frequency value and a predetermined second formant frequency value. The color according to the combination of the specified first formant frequency value and the specified second formant frequency value is a predetermined first formant frequency value and the specified first formant frequency. The greater the sum of squares of the difference between the two values and the difference between the predetermined second formant frequency value and the specified second formant frequency value, the greater the degree of change from the reference color.

  According to an eighth aspect of the present invention, in the display control device according to the sixth aspect, the storage means at least as the pronunciation feature amount specified for each predetermined time unit based on the first speech waveform data. A formant frequency, a second formant frequency, and a noise component are stored, and the specifying means at least the first formant frequency and the second formant as the pronunciation feature amount for each predetermined time unit based on the second speech waveform data. A frequency and a noise component are specified, and the third determining unit determines the color according to a combination of the specified first formant frequency value and the specified second formant frequency value, and the specified A mixed color with a color corresponding to a value of a noise component is determined as the color of the line.

  According to a ninth aspect of the present invention, in the display control apparatus according to the third aspect, the predetermined unit is a predetermined time unit, and the storage unit is configured to perform a predetermined time unit based on the first speech waveform data. At least a first formant frequency and a second formant frequency are stored as the pronunciation feature quantity specified in step (b), and the specifying means at least as the pronunciation feature quantity for each predetermined time unit based on the second speech waveform data. A first formant frequency and a second formant frequency are specified, and the third determining unit determines a color according to the value of the specified first formant frequency and a color according to the value of the specified second formant frequency. Is determined as the color of the line.

  According to a tenth aspect of the present invention, in the display control device according to the ninth aspect, the storage means at least as the pronunciation feature amount specified for each predetermined time unit based on the first speech waveform data. A formant frequency, a second formant frequency, and a noise component are stored, and the specifying means at least the first formant frequency and the second formant as the pronunciation feature amount for each predetermined time unit based on the second speech waveform data. A frequency and a noise component are specified, and the third determination unit includes a mixed color of a color according to the value of the specified first formant frequency and a color according to the value of the specified second formant frequency. A mixed color with a color corresponding to the value of the noise component is determined as the color of the line.

  The invention according to claim 11 is a display control method executed by one or more computers, and includes text data of a character string composed of a plurality of characters, and a model for reading the character string aloud. A storage step for storing in the storage means the pitch, sound pressure, and pronunciation feature amount specified for each predetermined unit based on the first speech waveform data indicating the waveform of the voice, and the speaker reads the character string aloud An input step for inputting second voice waveform data indicating the waveform of the voice uttered at the time, and a specification for specifying a pitch, a sound pressure, and a pronunciation feature amount for each predetermined unit based on the second voice waveform data And a time series that extends in the time axis direction and changes in a time series in an axis direction orthogonal to the time axis direction, based on the pitch, the sound pressure, and the sound generation feature amount stored in the storing step. 1 line screen A first control step of displaying, a second control step of displaying each character constituting the character string on the screen along the time axis direction based on the text data, and the sound specified by the specifying step A second line that extends in the time axis direction and changes in a time series in an axis direction orthogonal to the time axis direction based on high, sound pressure, and the sound generation feature amount, is the first line. A third control step for displaying on the screen in a comparable manner, wherein the first control step and the third control step are each based on the pitch and in the axial direction orthogonal to the time axis direction. Determining a line position for each predetermined unit; determining a line width for each predetermined unit based on the sound pressure; and Characterized in that it comprises the steps of determining emissions of colors for each of the predetermined unit, the.

  The invention according to claim 12 is based on text data of a character string composed of a plurality of characters and first sound waveform data indicating a sound waveform serving as a model when the character string is read aloud. A storage step for storing the pitch, sound pressure, and pronunciation feature quantity specified for each unit in the storage means, and second speech waveform data indicating a waveform of speech uttered when the speaker reads the character string aloud. An input step for inputting, a specifying step for specifying a pitch, a sound pressure, and a pronunciation feature amount for each predetermined unit based on the second speech waveform data; a pitch, a sound pressure stored by the storing step; And a first control step for displaying, on the screen, a first line extending in a time axis direction and changing in a time series in an axis direction orthogonal to the time axis direction based on the pronunciation feature amount, and the text Based on data Based on the second control step of displaying each character constituting the character string on the screen along the time axis direction, the pitch, the sound pressure, and the pronunciation feature amount specified by the specifying step, A third control step for displaying a second line extending in the axial direction and changing in time series in an axial direction orthogonal to the time axis direction on the screen in a manner comparable to the first line; And causing the computer to execute the first control step and the third control step, respectively, for determining the position of the line in the axial direction orthogonal to the time axis direction for each predetermined unit based on the pitch. And determining the width of the line for each predetermined unit based on the sound pressure, and determining the color of the line for each predetermined unit based on the pronunciation feature amount. , Characterized in that it comprises a.

  According to the first, third, eleventh, and twelfth aspects of the present invention, the position, width, and color of the first line that represents a change in voice that serves as a model when reading a character string aloud are set to the first voice waveform data. The position, width, and color of the second line that represents a change in speech that is produced when the speaker reads the character string aloud are determined based on the pitch, sound pressure, and pronunciation feature amount specified from It is determined based on the pitch, sound pressure, and pronunciation feature quantity specified from the second speech waveform data, and the first line and the second line are displayed on the screen so that they can be compared. Since it is configured to be displayed on the screen along the time axis direction, it is possible to make the speaker understand the difference in how the voice is used as an example and the voice of the speaker at a glance. .

  According to the invention described in claim 2, since the position and size of the character to be displayed are determined based on the pitch and the sound pressure, the voice of the model and the voice of the speaker are emitted. This makes it possible for the speaker to understand the difference in direction more easily.

  According to the fourth aspect of the invention, since the color corresponding to each vowel specified for each character unit is determined as the line color, the voice of the model and the voice of the speaker It is possible to make the speaker clearly grasp the difference in how to utter each character.

  According to the fifth aspect of the present invention, since the line color variation can be increased, the visual effect given to the speaker can be improved.

  According to the invention described in claim 6, since the color corresponding to the combination of the value of the first formant frequency and the value of the second formant frequency specified every predetermined time is determined as the line color. The color of the line can be changed smoothly over time, allowing the speaker to understand the difference between the voice of the model and the speaker's voice even more clearly. be able to.

  According to the seventh and eighth aspects of the invention, the line color variation can be increased, so that the visual effect given to the speaker can be improved.

  According to the ninth aspect of the present invention, the mixed color of the color corresponding to the value of the first formant frequency specified every predetermined time and the color corresponding to the value of the second formant frequency is determined as the line color. Because the line color can be changed smoothly over time, the difference between the voice of the model and the voice of the speaker can be further improved for the speaker. Can be understood easily.

  According to the tenth aspect of the present invention, since the line color variation can be increased, the visual effect given to the speaker can be improved.

It is a figure which shows the schematic structural example of the audio | voice information display apparatus S which concerns on this embodiment. It is a figure which shows an example of a formant distribution map. It is a figure which shows an example of the example audio | voice line and speaker audio | voice line which are displayed on a screen during a speaker's reading aloud. It is a figure which shows the data used by the flow of a process in the audio | voice information display apparatus S, and a process. It is a figure which shows an example of the audio | voice drawing data production | generation processing content shown in FIG. It is a figure which shows an example of the screen drawing process content shown in FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, embodiment described below is embodiment at the time of applying this invention to an audio | voice information display apparatus.

[1. Configuration and function of the voice information display device S]
First, the configuration and function of the audio information display device S according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram illustrating a schematic configuration example of the audio information display device S according to the present embodiment. Note that examples of the voice information display device include a personal computer and a portable information terminal (smart phone or the like). As shown in FIG. 1, the audio information display device S includes a communication unit 1, a storage unit 2, a control unit 3, an operation unit 4, an interface (IF) unit 5, and the like. 6 is connected. The operation unit 4 receives an operation instruction from the user and outputs a signal corresponding to the received operation to the control unit 3. The interface unit 5 is connected to a microphone M, a display D, and the like. The microphone M collects a sound produced when a speaker who performs language learning, utterance utterance training, or the like reads a character string composed of a plurality of characters (for example, an announced character string). The display D displays voice information to be provided to the speaker on the screen according to a drawing command from the control unit 3. The voice information is information including a voice line representing a change in voice and the character string. The microphone M and the display D may be integrated with the audio information display device S or may be separate.

  The communication unit 1 communicates with a server or the like by connecting to a network (not shown) by wire or wireless. The storage unit 2 includes, for example, a hard disk drive and stores an OS (Operating System), a display control processing program (an example of the program of the present invention), and the like. The display control processing program is a program that causes the control unit 3 as a computer to execute display control processing described later. The display control processing program may be downloaded from a predetermined server as an application, or may be provided by being stored in a recording medium such as a CD or a DVD. The storage unit 2 also stores text data of a character string composed of a plurality of characters and first sound waveform data (hereinafter referred to as “example sound”) indicating a sound waveform as a model when the character string is read aloud. Waveform data ”). Here, the text data includes, for example, the sound generation timing of each character (for example, the elapsed time from the start of sound generation) in association with each character. Examples of character strings to be read aloud include, for example, character strings used in language learning or announcement training, or character strings used in singing. The model voice waveform data is stored in a predetermined voice file format.

  The control unit 3 includes a CPU (Center Processing Unit) as a computer, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 3 functions as an audio processing unit 31 and a drawing control unit 32 by a display control processing program. The voice processing unit 31 is an example of an input unit and a specifying unit in the present invention. The drawing control unit 32 is an example of a first control unit, a second control unit, and a third control unit in the present invention. The RAM in the storage unit 2 or the control unit 3 is an example of a storage unit in the present invention.

  The voice processing unit 31 inputs model voice waveform data stored in a predetermined voice file format from the storage unit 2. The voice processing unit 31 also outputs second voice waveform data (hereinafter referred to as “speaker voice waveform”) indicating a waveform of a voice collected when the speaker reads the character string and collected by the microphone M. Data). The model voice waveform data and the speaker voice waveform data are collectively referred to as voice waveform data. The audio waveform data is discrete time-series sound pressure waveform data, for example, sampling rate of 44.1 kHz, quantization of 16 bits, and monaural waveform data. Then, the voice processing unit 31 specifies the pitch, the sound pressure, and the pronunciation feature amount for each predetermined unit based on the model voice waveform data. In addition, the voice processing unit 31 specifies a pitch, a sound pressure, and a pronunciation feature amount for each predetermined unit based on the speaker voice waveform data. Note that the predetermined unit may be a time unit (for example, 10 ms to 100 ms) or a character unit.

  Here, the pitch (pitch) refers to the pitch of the sound. For example, the speech processing unit 31 calculates a fundamental frequency (Hz) from data cut out from speech waveform data at predetermined time intervals, for example, and specifies (determines) the calculated fundamental frequency (Hz) at predetermined time intervals as a pitch. . The pitch data indicating the pitch specified in this way is stored every predetermined time. For the pitch calculation method, for example, a known method such as a zero cross method or vector autocorrelation can be applied.

  Next, the sound pressure refers to a change (Pa) in air pressure due to sound waves. In this embodiment, the sound pressure level (dB) that represents the magnitude of effective sound pressure (Pa), which is the root mean square (RMS) of instantaneous sound pressure (Pa), is expressed as a numerical value that is easy to handle in calculation. To do. The sound pressure level (dB) is also called volume in a broad sense. For example, the sound processing unit 31 calculates the sound pressure level (dB) from data cut out from the sound waveform data, for example, at predetermined time intervals, and specifies the calculated sound pressure level (dB) at predetermined time intervals as the sound pressure. Sound pressure data indicating the sound pressure specified in this way is stored every predetermined time.

  Next, the pronunciation feature value is a feature value that can be distinguished by the frequency spectrum of the speech waveform data. The frequency spectrum indicates, for example, a frequency spectrum when the frequency is taken on the horizontal axis (X axis) and the power of the sound at the frequency (for example, the square of the sound pressure level) is taken on the vertical axis (Y axis). An example of the pronunciation feature amount is a phoneme. One phoneme basically corresponds to one character. Examples of phonemes include three vowels only, consonants only, and combinations of consonants and vowels. There are five vowels: a (a), i (i), u (u), e (e), and o (o). Consonants include sound components other than vowels (for example, k, s, t, n, h, m, y, r, w...). In addition, for example, the phoneme “ka” in Japanese is “ka” in Roman notation, and is therefore a combination of consonants and vowels. Also, the phoneme “sha” (recognized as one character) in Japanese is “sya” in the Roman alphabet, so this is also a combination of a consonant and a vowel. In addition, since the phoneme “de” in Japanese is “de” in Roman notation, this is also a combination of consonants and vowels. Note that the Japanese phoneme “n” is “n” or “m” in Roman notation, so it is a consonant.

  When it is possible to clearly delimit between letters (in other words, between phonemes) from the speech waveform indicated by the speech waveform data, it is possible to identify (determine) vowel distinction or phoneme distinction as an example of pronunciation features It is. In this case, for example, the speech processing unit 31 calculates a frequency spectrum by performing Fourier analysis (FFT) on data cut out from speech waveform data, for example, for each character (in other words, for each phoneme). Then, the speech processing unit 31 compares the calculated frequency spectrum with a template indicating the frequency spectrum for each phoneme prepared in advance (template matching) to specify a vowel or phoneme as a pronunciation feature amount for each character. . The pronunciation feature quantity indicating the vowel or phoneme specified in this way is stored for each character. That is, for each character, a vowel distinction or a phoneme distinction is stored. In Japanese, there is a character “n” that consists only of consonants, so that “a”, “i”, “u”, “e”, “o”, “ It will be another. When specifying phonemes, it is desirable to perform a Fourier analysis by separating a speech waveform cut out for each character into a consonant part having a relatively small amplitude and a vowel part having a relatively large amplitude.

  Other examples of the pronunciation feature amount include formant frequency and noise frequency. The formant frequency is a frequency that becomes a peak in the spectrum envelope specified by the frequency spectrum, and is called the first formant frequency, the second formant frequency, the third formant frequency,... In general, the vowel distinction can be determined to some extent based on the formant distribution (two-dimensional coordinate plane) when the first formant frequency is taken on the horizontal axis and the second formant frequency is taken on the vertical axis. FIG. 2 shows an example of a formant distribution diagram. Note that the formant distribution varies depending on gender, language, and the like. In the two-dimensional plane of the formant distribution shown in FIG. 2, the region R1 indicates the region corresponding to the vowel “a”, the region R2 indicates the region corresponding to the vowel “i”, and the region R3 indicates the vowel. A region corresponding to “u” is shown, a region R4 indicates a region corresponding to the vowel “e”, and a region R5 indicates a region corresponding to the vowel “o”. For example, a combination in which the value of the first formant frequency is 1000 (Hz) and the value of the second formant frequency is 1200 (Hz) is in a region corresponding to the vowel “a”. In some cases, the combination of the value of the first formant frequency and the value of the second formant frequency is within a plurality of regions. The noise frequency refers to the frequency of noise components other than the fundamental frequency and frequencies other than harmonics thereof. Note that at the time when there is no clear fundamental frequency, the spectrum of the noise component is smoothed in the frequency axis direction, and the peak of the largest peak in the spectrum envelope is called the noise center frequency. Consonants contain a lot of noise frequencies.

  When the pronunciation feature amount is a formant frequency and a noise frequency, for example, the speech processing unit 31 performs Fourier analysis on the speech waveform data, separates the noise component and the harmonic component in frequency bin units, and separates the separated components. The speech waveform data of the noise component and the speech waveform data of the harmonic component are generated again by inverse Fourier analysis. Then, the voice processing unit 31 calculates the frequency spectrum of the noise component by performing Fourier analysis on data cut out from the voice waveform data of the noise component, for example, every predetermined time. Further, the sound processing unit 31 calculates a noise frequency by performing smoothing on the frequency axis in the frequency spectrum of the calculated noise component, and specifies the calculated noise frequency as a pronunciation feature amount at predetermined time intervals. The pronunciation feature quantity indicating the noise frequency specified in this way is stored every predetermined time. Since the spectrum envelope of the noise component often has a large mountain shape, the sound processing unit 31 specifies and stores the peak of the mountain as the noise center frequency. On the other hand, the voice processing unit 31 calculates the frequency spectrum of the harmonic component by performing Fourier analysis on data cut out from the voice waveform data of the harmonic component, for example, every predetermined time. Then, the sound processing unit 31 calculates a formant frequency from the calculated frequency spectrum of the harmonic component by a cepstrum method, and specifies the calculated formant frequency as a pronunciation feature amount at predetermined time intervals. Alternatively, the speech processing unit 31 calculates a formant frequency by using a linear predictive coding (LPC) method for data cut out from speech waveform data of harmonic components at predetermined time intervals, and calculates the calculated formant frequency. Is specified at predetermined time intervals as a pronunciation feature amount. The pronunciation feature amount indicating the formant frequency specified as described above is stored every predetermined time. The speech processing unit 31 specifies the first formant frequency and the second formant frequency, and stores the first peak and the second peak of the peak of the spectrum envelope obtained by the cepstrum method or the linear predictive coding method. . Further, in the above, the speech waveform data of the noise component generated again by the inverse Fourier analysis and the speech waveform data of the harmonic component may be synthesized and returned to the speech waveform data. In this case, sound pressure or the like may be specified based on the returned speech waveform data.

  In addition, the sound processing unit 31 calculates a sound pressure level (dB) from, for example, data cut out every predetermined time from the sound waveform data of the noise component generated as described above, and the calculated sound pressure level (dB) is calculated as noise. The sound pressure may be specified every predetermined time. Noise sound pressure data indicating the noise sound pressure specified in this way is stored every predetermined time. In addition, the sound processing unit 31 calculates a sound pressure level (dB) from, for example, data cut out every predetermined time from the sound waveform data of the harmonic component generated as described above, and calculates the calculated sound pressure level (dB). The harmonic sound pressure may be specified every predetermined time. The harmonic sound pressure data indicating the harmonic sound pressure specified in this way is stored every predetermined time.

  Next, the drawing control unit 32 extends in the time axis (for example, the horizontal axis) direction based on the pitch, sound pressure, and pronunciation feature amount specified from the model voice waveform data, and the time axis direction A first voice line (hereinafter referred to as “example voice line”) that changes in time series in an orthogonal axis (for example, vertical axis) direction is displayed on the screen of the display D. Here, when displaying the model voice line, the drawing control unit 32 displays the position of the model voice line in the axial direction orthogonal to the time axis direction (based on the pitch specified from the model voice waveform data ( Coordinate) is determined, for example, every predetermined time, and the width (line width) of the model voice line is determined, for example, every predetermined time based on the sound pressure specified from the model voice waveform data. Further, the drawing control unit 32 sets the color (line color) of the model voice line for each predetermined unit (for example, for each character or for each predetermined time) based on the pronunciation feature amount specified from the model voice waveform data. decide.

  Here, a specific example of a method for determining the color of an audio line will be described.

(1) Specific example 1 of audio line color determination method
For example, the drawing control unit 32 determines for each character so that the line color differs according to the specified vowel (that is, the color set in advance for each vowel is determined as the line color). Thereby, the speaker can be made to grasp clearly the difference in how to utter the voice as a model and the voice of the speaker for each character. Alternatively, the drawing control unit 32 may determine, for each character, a mixed color of a color corresponding to the specified vowel and a color corresponding to the specified consonant as a line color. Thereby, since the variation of the color of an audio | voice line can be increased, the visual effect given with respect to a speaker can be improved. In this case, a color other than the color set for the vowel is set in advance as the consonant.

(2) Specific example 2 of audio line color determination method
For example, the drawing control unit 32 determines a color corresponding to a combination of the specified first formant frequency value and the specified second formant frequency value as a line color at predetermined time intervals. As a result, the color of the voice line can be changed smoothly over time, and the difference between the voice of the model and the voice of the speaker can be further understood for the speaker. It can be easily grasped. In addition, even if it is difficult to clearly separate the characters from the speech waveform indicated by the speech waveform data, the speaker will be able to explain the difference in how the speech is used as an example and the speech of the speaker. On the other hand, it is possible to make it easier to understand. In this case, for example, a predetermined reference color is preset for each of a plurality of reference combinations of a predetermined first formant frequency value and a predetermined second formant frequency value. This reference combination is as many as the number of vowels, and it is desirable that there are different reference colors for each vowel. For example, in the two-dimensional plane of the formant distribution shown in FIG. 2, “blue” is preset as a predetermined reference color for the reference combination corresponding to the substantially central portion of the region R1 of the vowel “a”. This reference combination indicates (x1, y1) = (value of the first formant frequency, value of the second formant frequency) or indicates a group of coordinates in the region R1 including (x1, y1). Also, “red” is set in advance as a predetermined reference color for the reference combination corresponding to the substantially central portion of the region R2 of the vowel “i”. This reference combination indicates (x2, y2) = (value of the first formant frequency, value of the second formant frequency) or indicates a group of coordinates in the region R2 including (x2, y2). Further, “green” is set as the predetermined reference color for the reference combination corresponding to the substantially central portion of the region R3 of the vowel “u”. This reference combination indicates (x3, y3) = (value of the first formant frequency, value of the second formant frequency) or indicates a group of coordinates in the region R3 including (x3, y3). Further, “purple” is set as the predetermined reference color for the reference combination corresponding to the substantially central portion of the region R4 of the vowel “e”. This reference combination indicates (x4, y4) = (value of the first formant frequency, value of the second formant frequency) or indicates a group of coordinates in the region R4 including (x4, y4). Further, “yellow” is set as the predetermined reference color for the reference combination corresponding to the substantially central portion of the region R5 of the vowel “o”. This reference combination indicates (x5, y5) = (value of the first formant frequency, value of the second formant frequency) or indicates a group of coordinates in the region R5 including (x5, y5).

  Then, the drawing control unit 32 determines, for example, a reference having the closest distance on the two-dimensional plane from the combination (x0, y0) of the specified first formant frequency value and the specified second formant frequency value. A color (referred to as a harmonic color) preset for the combination (or a reference combination including the combination (x0, y0)) is determined as a line color every predetermined time. Alternatively, the drawing control unit 32 may select a color (harmonic color) corresponding to the combination of the specified first formant frequency value and the specified second formant frequency value and the specified noise frequency (for example, noise). A mixed color with a color corresponding to the value of the (center frequency) (which is an example of a color corresponding to the value of the noise component and is referred to as a noise color) may be determined as a line color every predetermined time. Thereby, since the variation of the color of an audio | voice line can be increased, the visual effect given with respect to a speaker can be improved. In this case, a color other than the reference color is set in advance as the value of the noise frequency (for example, the noise center frequency).

  Alternatively, the drawing control unit 32 increases the degree of change from the reference color set in advance for the reference combination as the distance on the coordinate plane increases from the reference combination (xα, yβ). You may determine the color (harmonic color) according to the combination of the value of the specified 1st formant frequency and the value of the specified 2nd formant frequency for every predetermined time. Thereby, since the variation of the color of an audio | voice line can be increased, the visual effect given with respect to a speaker can be improved. Here, (xα, yβ) represents (x1, y1), (x2, y2), (x3, y3), (x4, y4), or (x5, y5). For example, when (xα, yβ) is (x1, y1), the closer to the outer edge of the region R1 shown in FIG. 2, the more the reference color (for example, (R The degree of change from (luminance value, G luminance value, B luminance value) = (0,0,255)) increases. A large degree of change from the reference color means, for example, that the color difference from the reference color is large. For example, when the color is expressed in RGB, the color difference is obtained from the Euclidean distance from the coordinates (0, 0, 255) of the reference color in the RGB three-dimensional color space (for example, the color is expressed by 8 bits). In this case, the drawing control unit 32 determines the difference between the predetermined first formant frequency value xα and the specified first formant frequency value x0 and the predetermined second formant frequency value yβ. Calculate the sum of squares ((xα-x0) ^ 2 + (yβ-y0) ^ 2) with the difference between the values of y0 (or calculate the square root of the sum of squares). Then, the drawing control unit 32 specifies the value of the specified first formant frequency so that the degree of change from the reference color increases as the calculated square sum (or the square root of the square sum) increases. The color corresponding to the combination with the value of the two formant frequencies is determined by calculation. For example, the color according to the combination is obtained by adding a value (however, 255 or less) proportional to the sum of squares to the R luminance value (0) or G luminance value (0) in the reference color (0,0,255). It is determined by subtracting a value (however, 255 or less) proportional to the square sum from the B luminance value (255).

(3) Specific example 3 of voice line color determination method
For example, the drawing control unit 32 displays a mixed color of a color (harmonic color) corresponding to the specified first formant frequency value and a color (harmonic color) corresponding to the specified second formant frequency value. The line color is determined every predetermined time. This makes it possible to change the color of the line smoothly over time, making it easier for the speaker to understand the difference between the voice of the model and the voice of the speaker. It can be grasped. In addition, even if it is difficult to clearly separate the characters from the speech waveform indicated by the speech waveform data, the speaker will be able to explain the difference in how the speech is used as an example and the speech of the speaker. On the other hand, it is possible to make it easier to understand. For example, the drawing control unit 32 converts the value of the specified first formant frequency into a G luminance value of, for example, 255 or less (for example, according to a conversion formula of luminance value = frequency / k (coefficient)). The luminance value is determined as a color corresponding to the value of the first formant frequency. Further, the drawing control unit 32 converts the specified second formant frequency value into a B luminance value of, for example, 255 or less, thereby determining the B luminance value as a color corresponding to the second formant frequency value. Then, the drawing control unit 32 determines the mixed color (for example, (R luminance value, G luminance value, B luminance value)) obtained from the determined G luminance value and B luminance value as a line color. Here, the R luminance value that is the remaining luminance value may be any value from 0 to 255, for example. However, the drawing control unit 32 determines the harmonic sound pressure value (the harmonic sound pressure data described above) ( By converting the sound pressure level (dB) to an R luminance value of, for example, 255 or less, it is more effective to adjust the density of the mixed color obtained from the G luminance value and the B luminance value. Note that the combination of the luminance value of the first formant frequency value and the second formant frequency value is not a combination of the G luminance value and the B luminance value, but a combination of the B luminance value and the G luminance value, and the R luminance value. And G luminance values, R luminance values and B luminance values, B luminance values and R luminance values, or G luminance values and R luminance values. The remaining luminance value may be adjusted according to the value of the harmonic sound pressure.

  Alternatively, the drawing control unit 32 mixes a color (harmonic color) according to the value of the specified first formant frequency (harmonic color) and a color (harmonic color) according to the value of the specified second formant frequency ( For example, as described above (R luminance value, G luminance value, B luminance value)) and a color (noise color) corresponding to the value of the specified noise frequency (for example, noise center frequency) The color may be determined every predetermined time as a line color. Thereby, since the variation of the color of an audio | voice line can be increased, the visual effect given with respect to a speaker can be improved. In this case, a color other than the reference color is set in advance as the value of the noise frequency (for example, the noise center frequency). The drawing control unit 32 selects a color corresponding to the value of the noise frequency (for example, the noise center frequency) according to the value of the noise component sound pressure (sound pressure level (dB)) indicated by the noise component sound pressure data. It is more effective to adjust the density of.

  The color of the audio line may be determined based on the pronunciation feature amount by a method other than the determination method described above.

  In addition, the drawing control unit 32 acquires the text data of the character string from the storage unit 2 and, based on the acquired text data, moves each character constituting the character string along the time axis (for example, the horizontal axis) direction. Display on the screen of the display D so as to correspond to the model voice line. Here, when displaying the character, the drawing control unit 32 determines the position of the character for each character in the axial direction orthogonal to the time axis direction based on the pitch specified from the model voice waveform data. The character size is determined for each character based on the sound pressure specified from the sample speech waveform data.

  Further, the drawing control unit 32 extends in the time axis (for example, the horizontal axis) direction and is orthogonal to the time axis direction based on the pitch, sound pressure, and pronunciation feature quantity specified from the speaker voice waveform data. A second voice line (hereinafter referred to as “speaker voice line”) that changes in time series in the direction of the axis (for example, the vertical axis) is displayed on the screen of the display D so as to be comparable to the model voice line. Here, when displaying the speaker voice line, the drawing control unit 32 determines the position of the speaker voice line in the axial direction orthogonal to the time axis direction based on the pitch specified from the speaker voice waveform data. For example, it is determined every predetermined time, and based on the sound pressure specified from the speaker voice waveform data, the width of the speaker voice line is determined, for example, every predetermined time. Further, the drawing control unit 32 determines the color of the speaker voice line for each predetermined unit (for example, for each character or every predetermined time) based on the pronunciation feature amount specified from the speaker voice waveform data. A specific example of the method for determining the color of the speaker voice line is the same as the specific examples 1 to 3 of the method for determining the color of the model voice line described above.

  In addition, the drawing control unit 32 uses the display D so that each character constituting the character string corresponds to the speaker voice line along the time axis (for example, the horizontal axis) based on the acquired text data. Display on the screen. For example, the drawing control unit 32 determines the character position in the axial direction orthogonal to the time axis direction for each character based on the pitch specified from the speaker voice waveform data, and is specified from the speaker voice waveform data. Based on the sound pressure, the character size is determined for each character.

  FIG. 3 is a diagram illustrating an example of a model voice line and a speaker voice line displayed on the screen while the speaker is reading aloud. FIG. 3 shows an example of a screen when a phoneme is specified as the pronunciation feature quantity. In the screen shown in FIG. 3, a model voice line display unit 51, a speaker voice line display unit 52, and a phoneme / line color association display unit 53 are provided. Each of the model voice line display unit 51 and the speaker voice line display unit 52 is composed of coordinate planes in which the pitch is the vertical axis (Y axis) and the time is the horizontal axis (X axis). In the model voice line display unit 51, a model voice line 51a and each character 51b constituting a character string are displayed. Further, the speaker voice line display section 52 displays the speaker voice line 52a and the characters 52b constituting the character string. In addition, the time increment in the time axis (horizontal axis) direction of the model voice line display unit 51 and the time increment in the time axis (horizontal axis) direction of the speaker voice line display unit 52 seem to match. Thus, the speaker voice line 52a and the model voice line 51a can be displayed in a comparable manner.

  The phoneme / line color correspondence display section 53 displays the correspondence between phonemes (in this example, vowels and “n”) and line colors (colors of voice lines). The association between phonemes and line colors is preset. In this example, “blue”, “red”, “green”, “purple”, “yellow” for the vowels of “a”, “i”, “u”, “e”, “o”, respectively. Is preset. Further, “gray” (gray) is preset for “n”. Note that “grey” may be set in advance for consonants other than “n”, or a different line color may be set in advance for each consonant. For example, when the phoneme is a combination of a consonant and a vowel such as “ka”, for example, a mixed color obtained by synthesizing a line color preset for a consonant and a line color preset for a vowel Become. Note that the line color may not be preset for the consonant. In this case, for example, when the phoneme is a combination of a consonant and a vowel such as “ka”, for example, the line color set for the vowel is used. The current utterance position shown in FIG. 3 indicates the current reading position of the speaker who is reading the character string.

  In addition, the position (y coordinate) in the vertical axis (Y-axis) direction of each point constituting the model voice line 51a displayed on the model voice line display unit 51 is the pitch specified from the model voice waveform data. Is determined every predetermined time (for example, 10 ms) in the horizontal axis (X-axis) direction. In this case, as the pitch is higher, the positions of the points constituting the example voice line 51a are determined to be positions where the y-coordinate value is larger on the coordinate plane. The width of the model voice line 51a displayed on the model voice line display unit 51 is determined based on the sound pressure specified from the model voice waveform data for a predetermined time in the horizontal axis (X-axis) direction (for example, 10ms). In this case, the higher the sound pressure is, the wider the width of the model voice line 51a is determined. The width of the model voice line 51a is the width (that is, the thickness) in the direction orthogonal to the direction in which the model voice line 51a extends (advances). Moreover, you may comprise so that the audio | voice part 51a may not be displayed in the area where the specified sound pressure is below a threshold value. A section x3 shown in FIG. 3 is a section where speech is not performed, and the sound pressure is equal to or lower than the threshold value, so the model voice line 51a is not displayed. As described above, the model voice line 51a may be cut off in the middle. In particular, when the voice uttered by the speaker is low, the voice line 52a may be cut off. Note that the position and width of the speaker voice line 52a displayed on the speaker voice line display unit 52 is also based on the pitch and sound pressure specified from the speaker voice waveform data. It is determined by the same method.

  Further, the color (line color) of the model voice line 51a displayed on the model voice line display unit 51 is based on the phoneme specified from the model voice waveform data in the horizontal axis (X-axis) direction (character ( That is, it is determined for each phoneme). This is determined by the above-described specific example 1 of the method for determining the color of the audio line. For example, the line color of the section x1 corresponding to “go”, which is a combination of consonants and vowels, is a color set in advance for the consonant “g” (an example of a color corresponding to a sound component other than the vowels) and a vowel. It is a mixed color with a preset color (color corresponding to each vowel) for “o”. Further, for example, the line color of the section x2 corresponding to the vowel “A” is a color set in advance for the vowel “a” (a color corresponding to the vowel). The time length of the section for each character is longer than the predetermined time (for example, 10 ms). Further, the time lengths of the sections for each character are not necessarily the same, and as shown in FIG. 3, the time lengths of x1 (for example, 100 ms) and x2 (for example, 300 ms) are different. . The color of the speaker voice line 52a displayed on the speaker voice line display unit 52 is also determined by the same method as that of the model voice line 51a based on the phoneme specified from the speaker voice waveform data. .

  Further, the position (y coordinate) in the vertical axis (Y-axis) direction of each character 51b displayed on the model voice line display unit 51 is determined based on the pitch specified from the model voice waveform data. It is determined for each character in the (X-axis) direction. In this case, as the pitch is higher, the position of each character 51b is determined to be a position where the value of the y coordinate is larger on the coordinate plane. The pitch used here is, for example, an average value of a plurality of pitches included in a section (for example, x1) for each character. Further, the size of each character 51b displayed on the model voice line display unit 51 is determined for each character in the horizontal axis (X-axis) direction. In this case, the higher the sound pressure, the larger the size of each character 51b. The sound pressure used here is, for example, an average value of a plurality of sound pressures included in a section for each character (for example, x1). The position and size of each character 52b displayed on the speaker voice line display unit 52 is also the same method as each character 51b based on the pitch and sound pressure specified from the speaker voice waveform data. Determined by

  As described above, the model voice line 51a and each character 51b constituting the character string and the speaker voice line 52a and each character 52b constituting the character string are displayed so as to be comparable. Thus, it is possible to make the speaker understand the difference in how the speaker utters the voice more easily.

  In the example of FIG. 3, since the line color is determined based on the phoneme specified as the pronunciation feature amount, the color clearly changes at the boundary of each character section. However, when the formant frequency or the like is specified as the pronunciation feature amount, the colors of the model voice line 51a and the speaker voice line 52a are based on the specified formant frequency (or formant frequency and noise frequency), respectively. It is determined every predetermined time (for example, 10 ms) in the horizontal axis (X-axis) direction. For this reason, the colors of the model voice line 51a and the speaker voice line 52a change more smoothly than those shown in FIG. 3 (in other words, change like a gradation).

[2. Example of operation of voice information display device S]
Next, an example of the operation of the audio information display device S will be described with reference to FIGS. The operation example described below is an example in the case where a formant frequency or the like is specified as the pronunciation feature amount. FIG. 4 is a diagram showing a flow of processing in the voice information display device S and data used in the processing. FIG. 5 is a diagram showing an example of the voice drawing data generation processing content shown in FIG. FIG. 6 is a diagram illustrating an example of the contents of the screen drawing process illustrated in FIG.

  In FIG. 4, first, for example, when a speaker designates a desired voice file as a model of reading aloud through the operation unit 4, the control unit 3 executes a voice file input process and stores it in the designated voice file. The inputted model voice waveform data is input (step S1). Next, the voice processing unit 31 of the control unit 3 executes voice drawing data generation processing based on the input model voice waveform data (step S2). In the voice drawing data generation process, as shown in FIG. 5, a pitch data calculation process (step S21), a sound pressure data calculation process (step S22), and a pronunciation feature amount specifying process (step S23) are executed. The pitch data calculation process, the sound pressure data calculation process, and the pronunciation feature quantity specifying process may be executed in series or in parallel. When executed in series, the processing may be executed in any order among the pitch data calculation processing, the sound pressure data calculation processing, and the pronunciation feature amount identification processing.

  In the pitch data calculation process (step S21), the voice processing unit 31 executes a pitch specification process (step S211) for specifying the pitch at predetermined time intervals based on the input model voice waveform data. In the pitch specifying process, the voice processing unit 31 calculates, for example, a fundamental frequency (Hz) from data cut out from the model voice waveform data every predetermined time, and uses the calculated fundamental frequency (Hz) as a pitch for a predetermined time. Identify every time. Then, the voice processing unit 31 calculates pitch data indicating the pitch specified every predetermined time in time series.

  In the sound pressure data calculation process (step S22), the sound processing unit 31 executes a sound pressure specifying process (step S221) for specifying a sound pressure every predetermined time based on the input model sound waveform data. In the sound pressure specifying process, for example, the sound processing unit 31 calculates a sound pressure level (dB) from data cut out from the model sound waveform data at predetermined time intervals, and uses the calculated sound pressure level (dB) as the sound pressure. Identifies every predetermined time. Then, the sound processing unit 31 calculates sound pressure data indicating the sound pressure specified every predetermined time in time series.

  In the pronunciation feature amount specifying process (step S23), the speech processing unit 31 executes, for example, a noise / harmonic component separation process (step S231) for separating the noise component and the harmonic component in the input sample speech waveform data. . In the noise / harmonic component separation processing, for example, the speech processing unit 31 performs Fourier analysis on the sample speech waveform data, separates the noise component and the harmonic component in frequency bin units, and reverses each separated component again. Voice waveform data of noise components and voice waveform data of harmonic components are generated by Fourier analysis.

  Next, the sound processing unit 31 calculates a noise sound pressure specifying process (step S232) for specifying a noise sound pressure at every predetermined time and a noise center frequency at every predetermined time based on the generated sound waveform data of the noise component. The noise center frequency calculation process (step S233) is executed. In the noise sound pressure specifying process, the sound processing unit 31 calculates, for example, a sound pressure level (dB) from data cut out every predetermined time from the sound waveform data of the noise component, and uses the calculated sound pressure level (dB) as noise. The sound pressure is specified every predetermined time. Then, the sound processing unit 31 calculates noise sound pressure data indicating the noise sound pressure specified every predetermined time in time series. On the other hand, in the noise center frequency calculation process, the sound processing unit 31 calculates the frequency spectrum of the noise component by, for example, Fourier-analyzing data cut out from the sound waveform data of the noise component every predetermined time. Then, the audio processing unit 31 specifies the noise frequency every predetermined time by performing smoothing on the frequency axis in the frequency spectrum of the calculated noise component, and uses the apex as the noise center frequency in the spectrum envelope in time series. calculate.

  Further, the sound processing unit 31 calculates a harmonic sound pressure specifying process (step S234) for specifying the harmonic sound pressure for each predetermined time and a formant frequency for each predetermined time based on the generated sound waveform data of the harmonic component. The formant calculation process (step S235) is executed. In the harmonic sound pressure specifying process, the voice processing unit 31 calculates a sound pressure level (dB) from data cut out every predetermined time from the voice waveform data of the harmonic component, for example, and calculates the calculated sound pressure level (dB). The harmonic sound pressure is specified every predetermined time. Then, the voice processing unit 31 calculates harmonic sound pressure data indicating the harmonic sound pressure specified every predetermined time in time series. On the other hand, in the formant calculation process, the speech processing unit 31 calculates the first formant frequency and the second formant frequency using, for example, the linear predictive coding method for the data cut out from the harmonic waveform speech waveform data at predetermined time intervals. It is calculated in time series at every predetermined time.

  Then, the sound processing unit 31 performs the pitch data, the sound pressure data, the noise sound pressure data, the noise center frequency, the harmonic sound pressure data, the first formant frequency, and the second calculated by the sound drawing data generation process in step S2. As shown in FIG. 4, the formant frequency is stored as voice drawing data in, for example, a drawing DB (database) provided in the storage unit 2. Next, the control unit 3 executes a voice file input process, and inputs text data of a character string to be read aloud, for example, text data associated with a voice file designated by the speaker (step S3). 4), as shown in FIG. 4, it is stored in the drawing DB as text drawing data.

  On the other hand, when the speaker starts reading the character string, the voice generated during the reading of the character string is collected by the microphone M, and the speaker voice waveform data indicating the waveform of the collected voice is displayed in the interface unit. 5 to the audio information display device S. And the control part 3 inputs speaker audio | voice waveform data from the microphone M by performing an audio | voice input process (step S4). Next, the voice processing unit 31 of the control unit 3 executes the voice drawing data generation process shown in FIG. 5 based on the input speaker voice waveform data (step S5). In the voice drawing data generation process, the pitch data calculation process (step S21), the sound pressure data calculation process (step S22), and the pronunciation feature amount specification process (step S23) are performed in the same manner as in the case of the model voice waveform data. ) Is executed. Then, the sound processing unit 31 performs the pitch data, the sound pressure data, the noise sound pressure data, the noise center frequency, the harmonic sound pressure data, the first formant frequency, and the second calculated by the sound drawing data generation process in step S5. The formant frequency is sequentially output to the drawing control unit 32 as voice drawing data.

  Next, the drawing control unit 32 of the control unit 3 executes screen drawing processing (screen drawing processing for example) based on the voice drawing data (example) and text drawing data acquired from the drawing database, and the voice. Based on the voice drawing data (speaker) acquired from the processing unit 31 and the text drawing data acquired from the drawing database, a screen drawing process (screen drawing process for the speaker) is executed (step S6). Note that the model screen drawing process and the speaker screen drawing process may be executed in series or in parallel. Further, the screen drawing process for the speaker may be executed in real time while the speaker is reading aloud. In addition, although the example of FIG. 6 shows the content of one screen drawing process, this screen drawing process is performed for each of the model and the speaker.

  In the screen drawing process, as shown in FIG. 6, the drawing control unit 32 performs a text display process (step S61), a line display process (step S62), a line width change process (step S63), and a noise color generation process. (Step S64), harmonic color generation processing (Step S65), and color composition display processing (Step S66) are executed. The text display process and the line display process may be executed in series or in parallel. When executed in series, the process may be executed in any order among the text display process and the line display process.

  In the text display process (step S61), the drawing control unit 32 determines the display position of the display character (that is, the character to be displayed on the screen) and the display character in the horizontal axis (X axis) direction based on the acquired text drawing data. Decide for each character. The display position of the display character is determined by, for example, the sound generation timing included in the text drawing data. However, the display position of the character read aloud by the speaker is determined by, for example, a labeling process. The labeling process is based on text rendering data, speaker speech waveform data, and frequency spectrogram of speaker speech waveform data, and assigns phoneme labels according to the reading (utterance) content and specifies the boundary position between phonemes. It is a process to perform. Since various known methods can be applied to the labeling process, detailed description thereof is omitted. Next, the drawing control unit 32 determines the display position of the display character in the vertical axis (Y-axis) direction based on the pitch indicated by the pitch data included in the acquired voice drawing data, and the acquired voice drawing data The size of the display character is determined based on the sound pressure indicated by the included sound pressure data. And the drawing control part 32 draws a display character on a screen by giving a drawing command to the display D according to the determined display character, display position, and size.

  In the line display process (step S62), the drawing control unit 32, as described above, based on the pitch indicated by the pitch data included in the acquired voice drawing data, the voice line (Y axis) direction ( A display position of a model voice line or a speaker voice line is determined every predetermined time. In addition, the drawing control unit 32 identifies a section in which the sound pressure indicated by the sound pressure data included in the acquired voice drawing data is equal to or less than a threshold value. And the drawing control part 32 draws the audio | voice line which changes to a screen in time series by giving a drawing command to the display D according to the determined display position. If a section where the sound pressure is equal to or less than the threshold is specified, the drawing control unit 32 does not draw a voice line in this section.

  In the line width changing process (step S63), the drawing control unit 32 determines the width of the audio line at predetermined time intervals as described above based on the sound pressure indicated by the sound pressure data included in the acquired audio drawing data. To do. Then, the drawing control unit 32 changes the width of the audio line by giving a drawing command to the display D according to the determined width.

  In the noise color generation process (step S64), for example, the drawing control unit 32 determines a color according to the value of the noise center frequency included in the acquired voice drawing data, and uses the determined color density as the voice drawing data. The noise color is generated every predetermined time by determining according to the value of the noise sound pressure indicated by the noise sound pressure data included in the.

  In the harmonic color generation process (step S65), the drawing control unit 32, for example, the color according to the value of the first formant frequency included in the acquired voice drawing data and the second formant frequency included in the voice drawing data. Determine the mixed color with the color corresponding to the value, and determine the harmonic color by determining the darkness of the determined mixed color according to the harmonic sound pressure value indicated by the harmonic sound pressure data included in the voice drawing data Generate every hour.

  In the color composition display process (step S66), the drawing control unit 32 synthesizes (mixes) the noise color generated by the noise color generation process and the harmonic color generated by the harmonic color generation process to obtain a line color. Is determined every predetermined time. Then, the drawing control unit 32 gives a color to the audio line by giving a drawing command to the display D according to the determined line color.

  As described above, according to the above embodiment, the position, width, and color of the model voice line representing the change of the model voice when reading the character string is specified from the model voice waveform data. Speaker voice waveform indicating the position, width, and color of the speaker voice line, which is determined based on the pitch, sound pressure, and pronunciation feature amount, and represents the change in voice produced when the speaker reads the character string aloud. Based on the pitch, sound pressure, and pronunciation feature amount specified from the data, the model voice line and the speaker voice line are displayed on the screen so that they can be compared. Since it is configured to be displayed on the screen along the way, it is possible to make the speaker understand the difference between the voice of the example and the voice of the speaker at a glance.

DESCRIPTION OF SYMBOLS 1 Communication part 2 Memory | storage part 3 Control part 4 Operation part 5 Interface part 6 Bus 31 Voice processing part 32 Drawing control part S Voice information display apparatus

Claims (12)

Pitch and sound specified for each predetermined unit based on text data of a character string composed of a plurality of characters and first sound waveform data indicating a sound waveform serving as a model when the character string is read aloud Storage means for storing pressure and pronunciation features;
Input means for inputting second speech waveform data indicating a waveform of a speech uttered when a speaker reads out the character string;
A specifying means for specifying a pitch, a sound pressure, and a pronunciation feature amount for each predetermined unit based on the first speech waveform data;
A first line that extends in the time axis direction and changes in a time series in an axis direction orthogonal to the time axis direction based on the pitch, sound pressure, and the sound generation feature value stored in the storage means. First control means for displaying on the screen;
Second control means for displaying each character constituting the character string on the screen along the time axis direction based on the text data;
A second line that extends in the time axis direction and changes in time series in the axis direction orthogonal to the time axis direction based on the pitch, sound pressure, and sound generation feature amount specified by the specifying means. Is displayed on the screen so as to be comparable with the first line,
With
The first control means and the third control means are respectively
A first determining unit that determines the position of the line in the axial direction orthogonal to the time axis direction for each of the predetermined units based on the pitch;
A second determining unit that determines the width of the line for each predetermined unit based on the sound pressure;
A third determining unit that determines the color of the line for each predetermined unit based on the pronunciation feature amount;
A display control apparatus comprising: The second control means includes
A fourth determining unit that determines the position of the character in the axial direction orthogonal to the time axis direction for each predetermined unit based on the pitch;
A fifth determining unit that determines the size of the character for each of the predetermined units based on the sound pressure;
The display control apparatus according to claim 1, further comprising: The pronunciation feature amount is a feature amount distinguishable by a spectrum of the speech waveform data,
The display control apparatus according to claim 1, wherein the third determination unit determines a color of the line according to the pronunciation feature amount. The predetermined unit is a character unit,
The storage means stores a vowel as the pronunciation feature amount specified for each character unit based on the first speech waveform data,
The specifying means specifies a vowel as the pronunciation feature amount for each character unit based on the second speech waveform data,
The display control apparatus according to claim 3, wherein the third determination unit determines a color according to the specified vowel as the color of the line. The storage means stores a vowel and a sound component other than the vowel as the pronunciation feature amount specified for each character unit based on the first speech waveform data;
The specifying means specifies a vowel and a sound component other than the vowel as the pronunciation feature amount for each character unit based on the second speech waveform data;
The said 3rd determination part determines the mixed color of the color according to the specified said vowel and the color according to sound components other than the said vowel as the color of the said line. The display control apparatus described. The predetermined unit is a predetermined time unit,
The storage means stores at least a first formant frequency and a second formant frequency as the pronunciation feature amount specified for each predetermined time unit based on the first speech waveform data,
The specifying means specifies at least a first formant frequency and a second formant frequency as the sounding feature amount for each predetermined time unit based on the second speech waveform data;
The said 3rd determination part determines the color according to the combination of the value of the specified said 1st formant frequency and the value of the specified said 2nd formant frequency as a color of the said line. 4. The display control device according to 3. A predetermined reference color is set for a combination of a predetermined first formant frequency value and a predetermined second formant frequency value;
The color corresponding to the combination of the value of the specified first formant frequency and the value of the specified second formant frequency is the value of the predetermined first formant frequency and the value of the specified first formant frequency. 7. The degree of change from the reference color increases as the sum of squares of the difference and the difference between a value of a predetermined second formant frequency and the value of the specified second formant frequency increases. The display control apparatus described. The storage means stores at least a first formant frequency, a second formant frequency, and a noise component as the pronunciation feature amount specified for each predetermined time unit based on the first speech waveform data;
The specifying means specifies at least a first formant frequency, a second formant frequency, and a noise component as the pronunciation feature amount for each predetermined time unit based on the second speech waveform data;
The third determining unit includes a color corresponding to a combination of the specified first formant frequency value and the specified second formant frequency value, and a color corresponding to the specified noise component value. The display control apparatus according to claim 6, wherein the mixed color is determined as the color of the line. The predetermined unit is a predetermined time unit,
The storage means stores at least a first formant frequency and a second formant frequency as the pronunciation feature amount specified for each predetermined time unit based on the first speech waveform data,
The specifying means specifies at least a first formant frequency and a second formant frequency as the sounding feature amount for each predetermined time unit based on the second speech waveform data;
The third determining unit determines a mixed color of a color according to the value of the specified first formant frequency and a color according to the value of the specified second formant frequency as the color of the line. The display control apparatus according to claim 3, wherein The storage means stores at least a first formant frequency, a second formant frequency, and a noise component as the pronunciation feature amount specified for each predetermined time unit based on the first speech waveform data;
The specifying means specifies at least a first formant frequency, a second formant frequency, and a noise component as the pronunciation feature amount for each predetermined time unit based on the second speech waveform data;
The third determining unit is responsive to a mixed color of a color corresponding to the value of the specified first formant frequency and a color corresponding to the value of the specified second formant frequency, and a value of the noise component. The display control apparatus according to claim 9, wherein a mixed color with a color is determined as a color of the line. A display control method executed by one or more computers,
A pitch specified for each predetermined unit based on text data of a character string composed of a plurality of characters and first sound waveform data indicating a sound waveform serving as a model when the character string is read aloud, A storage step of storing the sound pressure and the pronunciation feature quantity in the storage means;
An input step of inputting second voice waveform data indicating a waveform of a voice uttered when the speaker reads out the character string;
A specifying step of specifying a pitch, a sound pressure, and a pronunciation feature amount for each predetermined unit based on the second speech waveform data;
A first line that extends in the time axis direction and changes in time series in an axis direction orthogonal to the time axis direction based on the pitch, sound pressure, and the sound generation feature value stored in the storage step. A first control step for displaying on the screen;
A second control step of causing each character constituting the character string to be displayed on the screen along the time axis direction based on the text data;
A second line extending in the time axis direction and time-sequentially changing in the axial direction orthogonal to the time axis direction based on the pitch, sound pressure, and the sound generation feature amount specified in the specifying step. Is displayed on the screen in a manner comparable to the first line, and
With
The first control step and the third control step are respectively
Determining the position of the line in the axial direction orthogonal to the time axis direction based on the pitch for each of the predetermined units;
Determining the width of the line for each predetermined unit based on the sound pressure;
Determining the color of the line for each predetermined unit based on the pronunciation feature amount;
A display control method comprising: A pitch specified for each predetermined unit based on text data of a character string composed of a plurality of characters and first sound waveform data indicating a sound waveform serving as a model when the character string is read aloud, A storage step of storing the sound pressure and the pronunciation feature quantity in the storage means;
An input step of inputting second voice waveform data indicating a waveform of a voice uttered when the speaker reads out the character string;
A specifying step of specifying a pitch, a sound pressure, and a pronunciation feature amount for each predetermined unit based on the second speech waveform data;
A first line that extends in the time axis direction and changes in time series in an axis direction orthogonal to the time axis direction based on the pitch, sound pressure, and the sound generation feature value stored in the storage step. A first control step for displaying on the screen;
A second control step of causing each character constituting the character string to be displayed on the screen along the time axis direction based on the text data;
A second line extending in the time axis direction and time-sequentially changing in the axial direction orthogonal to the time axis direction based on the pitch, sound pressure, and the sound generation feature amount specified in the specifying step. Is displayed on the screen in a manner comparable to the first line, and
To the computer,
The first control step and the third control step are respectively
Determining the position of the line in the axial direction orthogonal to the time axis direction based on the pitch for each of the predetermined units;
Determining the width of the line for each predetermined unit based on the sound pressure;
Determining the color of the line for each predetermined unit based on the pronunciation feature amount;
The program characterized by including.