JP6435791B2 - Display control apparatus and display control method - Google Patents

Description

  The present invention relates to a technique for displaying synthesis information used for speech synthesis.

  Conventionally, a speech synthesis technique for synthesizing a desired speech according to synthesis information designating a pronunciation character, a pronunciation period, and a pitch for each note has been proposed. For example, Patent Document 1 displays a music information image (a piano roll type image area in which a pitch axis corresponding to a pitch and a time axis corresponding to time) are set, and the user can display each musical note. A configuration is disclosed in which music information can be generated or edited while visually confirming a pitch, a pronunciation character, and a pronunciation period (a start point and an end point of a pronunciation period, and a duration).

JP 2011-095396 A

  By the way, a vowel is pronounced at the start of the note generation period specified by the synthesis information, and the consonant pronunciation period precedes the start of the note generation period. However, the technique of Patent Document 1 has a problem in that the user cannot visually recognize the starting point of the pronunciation period of the consonant because the pronunciation period specified by the synthesis information is only displayed on the music information image. It was. In view of the above circumstances, it is an object of the present invention to enable a user to visually grasp the starting point of the pronunciation period of a phoneme that is pronounced prior to the starting point of the pronunciation period specified by the synthesis information. And

  In order to solve the above-described problems, the display control apparatus of the present invention refers to the synthesis information for speech synthesis that specifies the pronunciation character, the pronunciation period, and the pitch for each note, and represents a note image representing each note. Is displayed on a time axis on a display device, and the display control means includes a first phoneme and the first phonetic character specified by the synthesis information. A second phoneme behind the phoneme, and in the synthesized speech to which the synthesis information is applied, the starting point of the pronunciation of the first phoneme precedes the starting point of the pronunciation period of the one note on the time axis In addition, a preceding phoneme image showing the starting point of the pronunciation of the first phoneme on the time axis is displayed in association with the note image of the one note. In the above configuration, the pronunciation character of one note specified by the synthesis information includes the first phoneme and the second phoneme, and the starting point of the first phoneme is the starting point of the pronunciation period of the one note. On the other hand, if the preceding phoneme image is preceded on the time axis, the preceding phoneme image is displayed, so that the user can visually grasp the starting point of the note pronunciation period from the note image, and can also determine the starting point of the first phoneme pronunciation. It is possible to visually grasp the preceding phoneme image. Note that the first phoneme is one phoneme immediately before the second phoneme, or any one of a plurality of phonemes preceding the second phoneme (typically, the first phoneme of the plurality). Means.

  In a preferred aspect of the present invention, the display control means displays a linear pitch transition image indicating the pitch transition of each note and the preceding phoneme image in association with the note image of each note, and The preceding phoneme image is a linear image that continues from the pitch transition image with the starting point of the first phoneme on the time axis as the end point. In the above aspect, the linear pitch transition image continuous to the pitch transition image with the start point of the first phoneme pronunciation as an end point is associated with the linear pitch transition image indicating the pitch transition of each note. Displayed. According to the above aspect, the user visually grasps the pitch transition of each note on the time axis from the pitch transition image, and visually identifies the starting point of the pronunciation of the first phoneme at the end point of the preceding phoneme image. It is possible to grasp. In addition, since the preceding phoneme image and the pitch transition image are displayed as linear images that are continuous on the time axis, the user can intuitively know the starting point of the first phoneme and the pitch transition of each note. It is possible to grasp it.

  In a preferred aspect of the present invention, the display control means changes the position of the end point of the preceding phoneme image on the pitch axis according to the pitch of the immediately preceding note. In the above aspect, the position of the end point of the preceding phoneme image on the pitch axis changes according to the pitch of the immediately preceding note. Therefore, by visually recognizing the preceding phoneme image, the user can visually grasp the starting point of the pronunciation of the first phoneme as described above, and can intuitively grasp the change in pitch from the immediately preceding note. There is.

  In a preferred aspect of the present invention, the display control means sets the position of the end point of the preceding phoneme image according to the pitch of the one note when the interval between the one note and the immediately preceding note exceeds a threshold value. When the interval between the one note and the immediately preceding note is less than a threshold value, the position of the end point of the preceding phoneme image is changed from the initial position. When there are successive notes on the time axis, the pitch of the preceding note tends to affect the pitch of the succeeding note according to the interval between the preceding note and the following note. Specifically, if the interval between the preceding note and the subsequent note is long, the pitch of the preceding note has less influence on the pitch of the subsequent note. On the other hand, when the interval between the preceding note and the subsequent note is short, the pitch of the preceding note has a great influence on the pitch of the subsequent note. According to the above aspect, the position of the end point of the preceding phoneme image is set to the initial position when the interval with the immediately preceding note exceeds the threshold value, and the position of the end point is changed from the initial position when the interval is less than the threshold value. There is an advantage that the user can intuitively grasp the influence of the pitch due to the interval.

  In a preferred aspect of the present invention, the display control means displays the pitch transition image and the preceding phoneme image in different aspects. As an example of a different aspect, for example, a configuration in which the color transition, the brightness, and the saturation of the pitch transition image and the preceding phoneme image are different from each other can be suitably employed. According to the above aspect, the user can visually and intuitively recognize the pitch transition of the note and the starting point of the pronunciation of the first phoneme.

  In a preferred aspect of the present invention, the display control means sets the time length of pronunciation of the first phoneme according to the type of the first phoneme. According to the above aspect, for example, the duration of pronunciation of the first phoneme is set according to the type of the first phoneme. Therefore, it is possible to set an optimum time length based on the difference between phoneme types.

  In a preferred aspect of the present invention, the synthesis information includes control information corresponding to an instruction from a user for each note, and the display control means uses the numerical value corresponding to the type of the first phoneme as an upper limit value. The time length of sound generation of one phoneme is set according to the control information. According to the above aspect, the time length of pronunciation of the first phoneme is set according to the control information designated by the user within the range of the upper limit value set for each phoneme type. It is possible to set a time length reflecting the user's instruction on the premise of the difference.

  The display control device according to each aspect described above is realized by hardware (electronic circuit) such as DSP (Digital Signal Processor) dedicated to editing synthesis information and generating audio signals, as well as a CPU (Central Processing Unit) It is also realized by cooperation between a general-purpose arithmetic processing device such as the above and a program. The program of the present invention can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium This type of recording medium can be included. For example, the program of the present invention can be provided in the form of distribution via a communication network and installed in a computer. The present invention is also specified as an operation method (display control method) of the display control device according to each aspect described above.

1 is a block diagram of a speech synthesizer 100 according to a first embodiment. It is a schematic diagram of the synthesis information S of the first embodiment. It is a schematic diagram of the edit screen 40 displayed on the display control part 24 of 1st Embodiment. It is a schematic diagram which expands and shows the note image 54 containing a 1st phoneme and a 2nd phoneme. It is explanatory drawing of the phoneme classification information F of 1st Embodiment. It is a flowchart which shows operation | movement of the display control part 24 of 1st Embodiment. It is a schematic diagram of the synthesis information S of the second embodiment. It is a schematic diagram of the edit screen 40 displayed on the display control part 24 of 2nd Embodiment. It is explanatory drawing of the phoneme classification information F of 2nd Embodiment. It is a schematic diagram of the edit screen 40 displayed on the display control part 24 of 3rd Embodiment. It is a schematic diagram of the edit screen 40 displayed on the display control part 24 of 3rd Embodiment. It is a schematic diagram of the edit screen 40 displayed on the display control part 24 of 4th Embodiment. It is a schematic diagram of the musical note iconic image 54 of a modification. It is a schematic diagram of the musical note iconic image 54 of a modification. It is a schematic diagram of the musical note iconic image 54 of a modification.

<First Embodiment>
FIG. 1 is a block diagram of a speech synthesizer 100 according to the first embodiment of the present invention. The speech synthesizer 100 is a signal processing device that generates speech such as speech sounds and singing sounds through segment-connected speech synthesis processing. As shown in FIG. 1, the arithmetic processing device 12, the storage device 14, and the display device are used. 16, an input device 17, and a sound emitting device 18. In 1st Embodiment, the case where the audio | voice signal Z of the singing voice of a specific music (henceforth "synthetic music") is produced | generated is assumed.

  The display device 16 (for example, a liquid crystal display device) displays an image instructed from the arithmetic processing device 12. The input device 17 is a device (for example, a mouse or a keyboard) that receives an instruction from a user. The sound emitting device 18 (for example, a headphone or a speaker) emits a sound wave corresponding to the sound signal Z generated by the arithmetic processing device 12.

  The storage device 14 stores a program PGM executed by the arithmetic processing device 12 and various data used by the arithmetic processing device 12. A known recording medium such as a semiconductor recording medium or a magnetic recording medium or a combination of a plurality of types of recording media is employed as the storage device 14. The storage device 14 of the first embodiment stores a speech unit group L, synthesis information S, and phoneme type information F as illustrated below.

  The speech segment group L is a set (speech synthesis library) of a plurality of speech segments collected in advance from recorded speech of a specific speaker. Each speech element is a phoneme unit (for example, a vowel or a consonant) that is the smallest unit of linguistic meaning, or a phoneme chain (for example, a diphone or a triphone) that connects a plurality of phonemes, and is a sample of a time domain speech waveform. It is expressed as a time series of a frequency domain spectrum calculated for each series or frame of a speech waveform.

  As illustrated in FIG. 2, the synthesis information S is time-series data that designates the singing voice of the synthesized music, and for each note constituting the synthesized music, the pronunciation character X1, the pronunciation period X2, and the pitch (for example, note number). Designate X3 in time series. The phonetic character X1 is a code that represents a syllable (mora) composed of a single vowel or a combination of consonants and vowels. The sound generation period X2 is the time length (note value) of a note, and is defined by, for example, the start time and time length (continuation length) or end time of sound generation. As can be understood from the above description, the synthesis information S can be rephrased as time-series data for designating the score of the synthesized music.

  The arithmetic processing unit 12 (CPU) has a plurality of functions (an information editing unit 22, a display control unit 24, a display control unit 24, and the like for generating a voice signal Z representing the waveform of the synthesized sound by executing the program PGM stored in the storage device 14. A speech synthesizer 28) is realized. A configuration in which each function of the arithmetic processing unit 12 is distributed over a plurality of integrated circuits, or a configuration in which a dedicated electronic circuit (DSP) realizes a part of the functions may be employed.

  The display control unit 24 displays various images on the display device 16. The display control unit 24 of the first embodiment refers to the synthesis information S and causes the display device 16 to display the editing screen 40 of FIG. 3 for the user to confirm and edit the content (note string) of the synthesized music. .

  FIG. 3 is a diagram illustrating an example of the editing screen 40. As illustrated in FIG. 3, the editing screen 40 includes a note string image N in which an image 54 representing each note instructed by the user (hereinafter referred to as “note image”) is arranged on the time axis. The note string image N includes a piano roll type coordinate plane in which a time axis (horizontal axis) and a pitch axis (vertical axis) intersecting each other are set. The position of the note image 54 in the direction of the pitch axis is selected according to the pitch X3 of the note, and the position and display length of the note image 54 in the direction of the time axis are selected according to the pronunciation period X2 of the note. . The user can instruct the addition of a new musical note graphic image 54 and the movement or expansion / contraction of the existing musical note graphic image 54 by appropriately operating the input device 17 while visually recognizing the editing screen 40.

  The information editing unit 22 in FIG. 1 manages the synthesis information S. Specifically, the information editing unit 22 generates and edits the composite information S in accordance with an instruction from the user to the input device 17. For example, the information editing unit 22 reflects the editing content on the editing screen 40 in accordance with an instruction to add a note image 54 to the note string image N, to move an arbitrary note image 54, or to expand or contract on the time axis. The composite information S is updated.

  The display control unit 24 causes the display device 16 to display the phonetic character X1 instructed by the user together with the musical note graphic image 54 (for example, superimposed on the musical note graphic image 54 as illustrated in FIG. 3). In the editing screen 40 shown in FIG. 3, a musical note image 54 in which each of the pronunciation characters X1 “su”, “na”, “a”, “ru”, “a” arranged on the time axis is assigned to five notes is displayed. Illustrated. As understood from the above examples, each phonetic character X1 is a phonetic character (“su [a]”) composed of a single vowel and a phonetic character (“su [su]” composed of a combination of a consonant and a vowel. ] ”,“ Na [na] ”,“ ru [ru] ”).

  The speech synthesizer 28 generates a speech signal Z by speech synthesis processing using the speech element group L and the synthesis information S stored in the storage device 14. The voice synthesis unit 28 of the first embodiment includes a pitch transition generation unit 282. The pitch transition generation unit 282 generates a change (hereinafter referred to as “pitch transition”) of the pitch of each note specified by the synthesis information S on the time axis. For example, the pitch transition generation unit 282 sets the pitch transition so that the pitch changes smoothly between successive notes on the time axis. The speech synthesizer 28 sequentially selects speech segments corresponding to each phonetic character X1 designated by the synthesis information S in time series from the speech segment group L, and the pitch transition generated by the pitch transition generator 282. The pitch of each speech segment is adjusted so as to extend along the line, and the speech signal Z is generated by expanding and contracting according to the sound generation period X2 and then connecting them. The speech synthesizer 28 synthesizes speech so that the vowel phonemes constituting the phonetic character X1 coincide with the start point of the note (start point of the note image 54). Specifically, for the phonetic character X1, which consists of a combination of consonants and vowels, the pronunciation of consonant phonemes begins before the beginning of each note's pronunciation period, and vowel pronunciation begins at the beginning of the note's pronunciation period. To synthesize speech. On the other hand, for the pronunciation character X1 composed of a single vowel, speech synthesis is performed so that the starting point of the vowel phoneme coincides with the starting point of the note.

  As illustrated in FIG. 3, the display control unit 24 according to the first embodiment displays each musical note image 54 in association with the transition image TR. As illustrated in FIG. 3, the display control unit 24 of the present embodiment includes a case where the phonetic character X1 is composed of a single vowel (“a [a]”) and a case where it is composed of a combination of consonants and vowels. (“Su [su]”, “na [na]”, “ru [ru]”), different transition images TR are displayed in association with the musical note image 54. Specifically, for example, when the phonetic character X1 is composed of a single vowel like the phonetic character “A [a]” in FIG. 3, the transition image TR is a sound generated by the pitch transition generation unit 282 for the note. It is composed of a linear pitch transition image PC showing a high transition (pitch curve). On the other hand, when the phonetic character X1 is composed of a combination of a consonant and a vowel (for example, the phonetic character “su [su]” including the first phoneme “s” of the consonant and the second phoneme “u” of the vowel). As illustrated in FIGS. 3 and 4, the transition image TR is composed of a preceding phoneme image G indicating the start point of the first phoneme on the time axis and a pitch transition image PC. The second phoneme is a phoneme behind the first phoneme (in the configuration in which the phonetic character X1 is composed of two phonemes of the first phoneme and the second phoneme as illustrated in the first embodiment, immediately after the first phoneme. Phoneme).

  FIG. 4 is an explanatory diagram of the transition image TR when the phonetic character X1 is composed of a combination of the first phoneme (consonant) and the second phoneme (vowel). The display control unit 24 associates the transition image TR including the preceding phoneme image G indicating the starting point of the pronunciation of the first phoneme “s” and the pitch transition image PC indicating the pitch transition of the note with the note image 54. To display. As illustrated in FIG. 4, the preceding phoneme image G is a linear image that continues from the subsequent pitch transition image PC with the start point of the pronunciation of the first phoneme “s” as the end point E. As described above, although the pitch cannot be actually specified for the consonant that is the first phoneme, in the transition image TR, for convenience, a linear shape in which the preceding phoneme image G and the pitch transition image PC are continuous on the time axis. Displayed as an image. In the above description, the phonetic character “su” displayed first in the note string image N of FIG. 3 has been described as an example, but other phonetics including the first phoneme and the second phoneme are described. For the characters (“na [na]” and “ru [ru]”), the display control unit 24 displays the transition image TR including the preceding phoneme image G and the pitch transition image PC.

  The display control unit 24 of the present embodiment displays the pitch transition image PC and the preceding phoneme image G constituting the transition image TR in different modes. For example, as illustrated in FIG. 4, the preceding phoneme image G and the pitch transition image PC are displayed by the display control unit 24 so that the thicknesses of the lines are different from each other. The user visually recognizes the preceding phoneme image G and the pitch transition image PC displayed in different modes on the editing screen 40, so that the first phoneme pronunciation start point E and each note on the time axis are displayed. It is possible to intuitively grasp both the pitch transition of the row image N.

  In FIG. 4, the duration TA defined by the end point E of the preceding phoneme image G and the start point of the note image 54 (the start point of the note generation) means the duration of the sound generation of the first phoneme “s”. On the other hand, the duration TB defined by the start point and the end point of the note image 54 means the time length of pronunciation of the second phoneme that is a vowel (that is, the pronunciation period X2). The first phoneme duration TA is set by the display control unit 24 according to the type of the first phoneme. For example, FIG. 3 illustrates a case where different durations TA are set according to the type of the first phoneme ([s], [n], and [r]). Phoneme type information F stored in the storage device 14 is used to set the phoneme duration TA for each type.

  FIG. 5 is an explanatory diagram of the phoneme type information F. The phoneme type information F is a data table that designates a duration TA (TA1, TA2,...) For each phoneme type. In FIG. 5, semi-vowels (/ w /, / y /), nasal sounds (/ m /, / n /), flow sounds (/ r /), burst sounds (/ t /, / k /, / p /), Rupture sound (/ ts /) and friction sound (/ s /, / f /) are illustrated as phoneme types. The pronunciation duration TA specified by the phoneme type information F is different for each phoneme type. For example, depending on the algorithm used for the decompression process, the time length TA of the rupture sound or the friction sound can be set to be longer than the time length TA of the plosive sound or the streaming sound.

  The display control unit 24 refers to the phoneme type information F, and uses the duration TA associated with the first phoneme type included in the phonetic character X1 of each note as the duration of the first phoneme pronunciation of the note. Set as length TA. In the example of FIG. 3, the first phoneme [s] of the first note has a duration TA6, the first phoneme [n] of the second note has a duration TA2 and the fourth note of the fourth note. For the first phoneme [r], the duration TA3 is set by the display control unit 24 as the duration of pronunciation TA of the first phoneme ([s], [n], [r]). The display control unit 24 displays the preceding phoneme image G over the duration TA from the start point of the note to the end point E by the duration TA set by the above method to the pitch transition image PC, and the note image 54 of each note. And the pitch transition image PC are displayed on the display device 16 in association with each other.

  FIG. 6 is a flowchart of a schematic operation of the display control unit 24 according to the first embodiment. For example, in the state in which the editing screen 40 is displayed on the display device 16 in response to an instruction from the user to the input device 17 (an instruction to edit the composite information S), an interrupt that occurs every predetermined time is used as an opportunity in FIG. Processing begins.

  The user appropriately operates the input device 17 while confirming the editing screen 40, thereby arranging the note image 54 at an arbitrary position of the note string image N to create a new note (hereinafter referred to as "target note"). It is possible to instruct addition and to specify the pronunciation character X1 of the target note. The display control unit 24 determines whether or not the user has instructed the addition of the target note by an operation on the input device 17 (SA1). When the target note is added (SA1: YES), the display control unit 24 determines whether the phonetic character X1 designated by the user for the target note includes the first phoneme and the second phoneme. (SA2). When the first phoneme and the second phoneme are included (SA2: YES), the display control unit 24 specifies the duration TA associated with the type of the first phoneme in the phoneme type information F, and the target note. The preceding phoneme image G is generated with the end point E as the end point E from the starting point by the duration TA (SA3). The display control unit 24 generates a pitch transition image PC indicating the pitch transition of each note generated by the pitch transition generation unit 282 (SA4), and the preceding phoneme image G of the first phoneme and the pitch transition image PC. Is displayed in association with the musical note iconic image 54 (SA5). On the other hand, when the pronunciation character X1 of the note is composed of, for example, a phoneme alone (SA2: No), the display control unit 24 converts the pitch transition image PC to the transition image without executing the process of generating the preceding phoneme image G. TR is displayed in association with the musical note image 54 (SA4-SA5).

  As understood from the above description, in the first embodiment, the synthesized character to which the pronunciation character X1 of the note designated by the synthesis information S includes the first phoneme and the second phoneme and to which the synthesis information S is applied. When the start point of the first phoneme is preceded by the start point of the note generation period (start point of the note image 54), the display control unit 24 displays the start phoneme image indicating the start point of the first phoneme on the time axis. G is displayed. Therefore, the user can visually grasp the starting point of the pronunciation of the first phoneme that is sounded before the starting point of the sounding period X2 of each note by the preceding phoneme image G.

  The preceding phoneme image G is configured as a linear image continuous with the pitch transition image PC with the starting point of the first phoneme pronunciation as an end point E, so that the user visually identifies the starting point E of the first phoneme pronunciation. While recognizing, it is possible to intuitively grasp the pitch transition of each note that is continuous on the time axis. In the first embodiment, since the preceding phoneme image G and the pitch transition image PC are displayed in different modes, the user clearly understands the pitch transition of the note and the starting point of the first phoneme. It is possible. In addition, since the display control unit 24 selects the duration of pronunciation of the first phoneme according to the type of the first phoneme (consonant type), the optimum duration TA based on the difference for each phoneme type is selected. It becomes possible to set.

Second Embodiment
A second embodiment of the present invention will be described below. In the first embodiment, the duration TA of the first phoneme of the phonetic character X1 is set to a fixed value specified from the phoneme type information F. In the second embodiment, the duration TA of the first phoneme is variably controlled according to an instruction from the user. In addition, about the element in which an effect | action and a function are the same as that of 1st Embodiment in each aspect illustrated below, the detailed description of each is abbreviate | omitted suitably using the code | symbol referred by description of 1st Embodiment.

  FIG. 7 is a schematic diagram of the synthesis information S of the second embodiment. As illustrated in FIG. 7, the synthesis information S of the second embodiment specifies control information V for each note in addition to the same information (phonetic character X1, pronunciation period X2, pitch X3) as in the first embodiment. To do. The control information V of the present embodiment is a parameter used for setting the continuation length TA, and is variably set according to an instruction from the user.

  FIG. 8 is an explanatory diagram for setting the control information V. The editing screen 40 in FIG. 8 is an image in which a control variable designation screen C is added below the note string image N as in the first embodiment. The control variable designation screen C is a screen for the user to designate the numerical value of the control information V for each note. FIG. 8 illustrates a case where the numerical value of the control information V of each note is displayed as a bar graph. The user can designate a desired value within a predetermined range for the control information V of each note by appropriately operating the input device 17. In the second embodiment, the pronunciation duration TA of the first phoneme is set according to the control information V.

  FIG. 9 is an explanatory diagram of the phoneme type information F of the second embodiment. The phoneme type information F includes an initial value L (L1, L2, L3, L4, L5, L6,...) And an upper limit A (A1, A2, A3, A4, A5, A6. This is a data table set for each type. Each of the initial value L and the upper limit value A is different for each phoneme type.

The display control unit 24 sets the continuation duration TA of the first phoneme included in the pronunciation character X1 of each note,
The initial value L and the upper limit value A corresponding to the type of the first phoneme are specified from the phoneme type information F, and a numerical value obtained by adjusting the initial value L according to the control information V within a range below the upper limit value A is Calculated as the duration TA of one phoneme. Specifically, FIG. 8 exemplifies a situation where the control information V of four notes for which a common phonetic character X1 is designated is set to different numerical values. As understood from FIG. 8, the initial value L is adjusted in accordance with the control information V within the range of the upper limit value A so that the continuation length TA decreases as the numerical value of the control information V increases. The display of the preceding phoneme image G corresponding to the continuation length TA is the same as in the first embodiment.

  In the second embodiment, the same effect as in the first embodiment is realized. In the second embodiment, since the duration TA of the first phoneme is variably set according to the control information V instructed by the user, the preceding phoneme image G reflecting the user's intention can be displayed. There is an advantage. Further, since the upper limit value A of the continuation length TA is set for each phoneme type, there is also an advantage that the preceding phoneme image G that visually represents the phoneme characteristics (long and short continuation length) can be displayed.

<Third Embodiment>
FIG. 10 is an explanatory diagram of an editing screen 40 displayed by the display control unit 24 of the third embodiment. As understood from FIG. 10, the display control unit 24 of the third embodiment changes the position of the end point E of the preceding phoneme image G on the pitch axis according to the pitch of the immediately preceding note. Specifically, the end point E of the preceding phoneme image G of any one note is higher on the high pitch side on the pitch axis as the pitch X3 of the immediately preceding note is higher than the pitch X3 of the note. It is located on the side near the pitch X3 of the immediately preceding note. For example, in FIG. 10, the display control unit 24 uses the second note and the fourth note to share the pitch X3, and the first note has a pitch X3 higher than that of the third note. It is assumed that the pitch difference between the first note and the second note is larger than the pitch difference between the third note and the fourth note. Therefore, as illustrated in FIG. 10, the display control unit 24 sets the end point E of the preceding phoneme image G associated with the second note image 54 in the note string image N to the fourth note image 54. Compared to the end point E of the preceding phoneme image G associated with, it is positioned at a higher position on the pitch axis.

  Further, the display control unit 24 determines that the end point E of the preceding phoneme image G of any one note is on the pitch axis as the pitch X3 of the immediately preceding note is lower than the pitch X3 of the note. It is displayed so as to be positioned on the low pitch side (the side close to the pitch X3 of the immediately preceding note). In FIG. 11, the display control unit 24 uses the second note and the fourth note to share the pitch X3, and the first note has a lower pitch X3 than the third note. It is assumed that the pitch difference between the first note and the second note is larger than the pitch difference between the third note and the fourth note. Therefore, as illustrated in FIG. 11, the display control unit 24 sets the end point E of the preceding phoneme image G associated with the second note image 54 to the preceding phoneme image associated with the fourth note image 54. Compared to the end point E of G, it is positioned at a lower position on the pitch axis.

  In the third embodiment, the same effect as in the first embodiment is realized. In the third embodiment, the position on the pitch axis of the end point E of the preceding phoneme image G changes according to the pitch X3 of the immediately preceding note. Therefore, there is an advantage that the user can intuitively grasp the transition (pitch difference) of the pitch from the immediately preceding note.

<Fourth embodiment>
In the third embodiment, the position on the pitch axis of the end point E of the preceding phoneme image G is changed according to the pitch X3 of the immediately preceding note, but there is a sufficient interval between the preceding note and the preceding note. It is assumed that there is no need to present to the user the correlation (pitch difference from the immediately preceding note) with the pitch of the immediately preceding note (or it is desirable not to present it dare). Considering the above circumstances, the display control unit 24 of the fourth embodiment determines whether or not to change the position of the end point E of the preceding phoneme image G according to the pitch X3 of the immediately preceding note. Switch according to the interval.

  FIG. 12 is an explanatory diagram of an edit screen 40 displayed by the display control unit 24 according to the fourth embodiment. In the note string image N of FIG. 12, the pronunciation characters “a [a]” of the first and third notes are composed of phonemes of a single vowel, and the pronunciations of the second and fourth notes are generated. The character “sa [sa]” is composed of a combination of consonants and vowels.

  When the section length TC of successive notes on the pitch axis exceeds the threshold value, the display control unit 24 positions the end point E of the preceding preceding phoneme image G at the initial position corresponding to the note pitch X3. Let When the section length TC between the backward note and the immediately preceding note is sufficiently long, it is considered that the correlation between the backward note and the immediately preceding note is low, and therefore, as illustrated in FIG. When the length TC2 exceeds the threshold Dth (TC2> Dth), the display control unit 24 sets the end point E of the preceding phoneme image G associated with the fourth note image 54 to the initial value corresponding to the pitch X3. Position (hereinafter referred to as “initial position”).

  On the other hand, when the section length TC of successive notes on the pitch axis is below the threshold, the display control unit 24 sets the position of the end point E of the preceding preceding phoneme image G in accordance with the pitch X3 of the note. Change from the initial position. When the section length TC between the backward note and the immediately preceding note is sufficiently short, it is considered that the correlation between the backward note and the immediately preceding note is high, and therefore, as illustrated in FIG. When the length TC1 is less than the threshold value Dth (TC1 <Dth), the display control unit 24 sets the end point E of the preceding phoneme image G associated with the second note image 54 to the pitch X3 of the immediately preceding note. Accordingly, for example, it is positioned on the extension line of the pitch transition image PC of the immediately preceding note. Therefore, there is an advantage that the user can visually and intuitively understand the influence of the pitch of each note due to the interval between successive notes (section length TC).

<Modification>
Each of the above-described embodiments can be variously modified. Specific modifications are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined.

(1) In each of the above-described embodiments, the linear preceding phoneme image G having the end point E as the starting point of the first phoneme is illustrated. However, the user can visually recognize the starting point E of the first phoneme. As long as it is possible, it is not limited to the above examples.

  FIG. 13 is an explanatory diagram of a preceding phoneme image G according to a modification. As illustrated in the area (a) of FIG. 13, the line type of the preceding phoneme image G is not limited to the above-described examples of each form. Further, as illustrated in the area (b) of FIG. 13, the preceding phoneme image G may be configured not as a linear image but as a dot image located at the starting point E of the first phoneme pronunciation. . Also according to the above aspect, the user can visually grasp the first phoneme pronunciation start point, and the first phoneme duration TA defined by the end point E and the note image start point 54. . As illustrated in the area (c) of FIG. 13, the preceding phoneme image G may be a rectangular figure that continues forward from the musical note image 54. As illustrated in the region (d) of FIG. 13, a line segment (for example, having a length equivalent to the height of the musical note image 54) arranged at the starting point of the first phoneme on the time axis and parallel to the pitch axis. It is also possible to display the line segment) as the preceding phoneme image G. Further, as understood from the regions (b) and (d) of FIG. 13, the preceding phoneme image G does not need to be continuous with the pitch transition image PC or the note image 54. In addition to the examples described above, for example, the preceding phoneme image G and the pitch transition image PC can be displayed in a common mode.

(2) In each of the embodiments described above, the start point of the first phoneme is the end point of the start point of the first phoneme that precedes the start point of the note period (start point of the note image 54) on the time axis. Although the linear preceding phoneme image G as E is illustrated, a configuration in which the subsequent phoneme image H following the end point of the second phoneme pronunciation (end point of the note image 54) is also displayed can be adopted. For example, depending on how speech segments are segmented and how the speech segments are expanded and contracted, there is a possibility that the speech segments are continuously pronounced until the end of the pronunciation period X2. Therefore, as illustrated in FIG. 14, the second phoneme aftertone that represents the end point of the pronunciation of the second phoneme of the pronunciation character “su [su]” (the end point of the note image 54) is followed. A phoneme image (that is, an image representing the pronunciation after the end of the pronunciation period X2) H may be displayed. Specifically, the succeeding phoneme image H is a linear image extending from the end point of the pitch transition image PC (end point of the sound generation period X2) to the end point of the sound segment.

(3) As illustrated in FIG. 15, an additional image B representing an additional pitch change such as vibrato other than pitch transition set from a time series of a plurality of notes, together with a pitch transition image PC It is also possible to display on the display device 16 in association with each musical note image 54.

(4) In each of the above-described embodiments, the pitch transition excluding an additional pitch change such as vibrato is represented by the pitch transition image PC, but the pitch specified only from the time series of a plurality of notes. It is also possible to represent a pitch transition including an additional pitch change (typically a pitch change as a singing expression) other than the change by a pitch transition image PC. As the additional pitch change, pitch bend and portamento (ascending / descending) can be exemplified in addition to the vibrato described above. The pitch transition image PC has a shape reflecting additional pitch change characteristics (parameters) such as the depth and speed of vibrato and pitch bend.

(5) In the second embodiment, the phoneme type information F in which the fixed value upper limit value A is associated with the initial value L of the continuation length TA is exemplified, and the initial value L is within the range of the upper limit value A. The configuration in which the duration TA of the first phoneme is variably set according to the instructed control information V has been exemplified. For example, a variable coefficient corresponding to the control information V with respect to the initial value L (hereinafter referred to as “expansion / contraction magnification”). In the configuration in which the duration TA is calculated by multiplying “)”, the maximum value of the expansion / contraction magnification can be set for each phoneme type. According to the configuration in which the initial value L is multiplied by the maximum magnification, even if the expansion / contraction magnification is common, the continuation length TA can be different depending on the continuation length of each phoneme of the speech segment, as in the second embodiment. Compared to the configuration in which the upper limit value A of the continuation length TA is set, the continuation length TA can be variously changed.

(6) In each of the above-described embodiments, the pitch transition image PC of the pitch transition generated by the pitch transition generating unit 282 of the voice synthesizing unit 28 is displayed. This is not always necessary for displaying the high transition image PC. In other words, the adjustment of the pitch of each speech unit to which the pitch transition generated by the pitch transition generation unit 282 is applied, the processing such as connection of each speech unit, and the like are only from the viewpoint of displaying the pitch transition image PC. If so, it can be omitted. According to the configuration in which processing necessary for generating the audio signal Z is omitted, there is an advantage that processing load (calculation time and the like) can be reduced.

(7) In each of the above-described embodiments, the preceding phoneme image G and the pitch transition image PC are displayed every time a user adds a note (target note). However, the trigger for updating the transition image TR is the above example (note Is not limited to). For example, when updating of the transition image TR of each note placed on the editing screen 40 is instructed by a user operation on the editing screen 40 (for example, an operation on the “redraw” button), Transition of each note automatically (for example, every predetermined time or every time the user instructs some editing) without adding or updating the transition image TR or an explicit instruction by the user A configuration for adding or updating the image TR may also be employed.

(8) In each of the above-described embodiments, the unit connection type speech synthesizer 28 using speech units is exemplified, but a known technique is arbitrarily employed for speech synthesis using the synthesis information S. For example, it is also possible to synthesize the singing voice of the synthesized music specified by the synthesis information S using a probabilistic model such as a Hidden Markov Model (HMM).

(9) In each of the above-described embodiments, the Japanese phonetic character X1 is exemplified, but the language of the phonetic character X1 (speech language to be synthesized) is arbitrary. For example, the present invention can be applied to the case of generating speech in an arbitrary language such as English, Spanish, Chinese, or Korean. Depending on the language, there are a plurality of consonants (first phonemes [s], [t], [r]) in front of the second phoneme [i] (vowel), for example, the English word “string”. The phonetic character X1 is also assumed. In this case, one consonant (typically, the first consonant [s] among the plural consonants ([s], [t], [r]) positioned in front of the second phoneme) is selected. As the first phoneme, it is also possible to display a preceding phoneme image G that represents the starting point of pronunciation of the first phoneme. As understood from the above description, the second phoneme is expressed as a phoneme located behind the first phoneme, and is not limited to the phoneme immediately after the first phoneme.

(10) In each of the above-described embodiments, the storage device 14 that stores the speech element group L and the synthesis information S is mounted on the speech synthesizer 100. However, an external device (for example, a server device) that is independent of the speech synthesizer 100. Is also used to store the speech element group L and the synthesis information S. The speech synthesizer 100 acquires the speech element group L or the synthesis information S via, for example, a communication network, and executes editing processing and speech synthesis processing. As understood from the above description, the elements that store the speech element group L and the synthesis information S are not essential elements of the speech synthesizer 100.

DESCRIPTION OF SYMBOLS 100 ... Voice synthesis device, 12 ... Arithmetic processing device, 14 ... Memory | storage device, 16 ... Display device, 17 ... Input device, 18 ... Sound emission device, 22 ... Information editing part, 24 ... Display Control unit 28... Speech synthesis unit 40 .. editing screen 54. Note image 282. Pitch transition generation unit G. Preceding phoneme image N. Note string image PC. Transition image, E ... End point, C ... Control variable designation screen.

Claims (4)

Referring to the synthesis information for speech synthesis that specifies the pronunciation character, the pronunciation period, and the pitch for each note, a note string image in which a note image representing each note is arranged on the time axis, and the note of each note Comprising display control means for displaying on a display device a linear pitch transition image that is arranged in association with a graphic image and indicates the pitch transition of each note .
The display control means includes a first phoneme and a second phoneme behind the first phoneme as a phonetic character specified by the synthesis information, and in the synthesized speech to which the synthesis information is applied. When the starting point of the first phoneme is preceded on the time axis with respect to the starting point of the sounding period of the one note, a linear shape that continues to the pitch transition image with the starting point of the first phoneme as the end point the preceding phoneme image is displayed in association with the musical note iconic image of a note of the one, the preceding phoneme image position on the pitch axes of said end points of the display control device Ru is changed according to the pitch of the note immediately preceding . Wherein the display control unit, the display control device according to claim 1 to set a time length of pronunciation of the first phoneme in accordance with the type of the first phoneme. The combined information, the display control means comprises for each note control information according to an instruction from the user, the upper limit value corresponding to the type of the first phoneme, the time length of pronunciation of the first phoneme the display control device according to claim 1 or claim 2 set according to the control information.   Referring to the synthesis information for speech synthesis that specifies the pronunciation character, the pronunciation period, and the pitch for each note, a note string image in which a note image representing each note is arranged on the time axis, and the note of each note A display control method for displaying on a display device a linear pitch transition image that is arranged in association with a graphic image and indicates a pitch transition of each note.
  The phonetic character of one note specified by the synthesis information includes a first phoneme and a second phoneme behind the first phoneme, and in the synthesized speech to which the synthesis information is applied, the pronunciation of the first phoneme When the start point precedes the start point of the sound generation period of the one note on the time axis, the linear preceding phoneme image continuous to the pitch transition image with the start point of the first phoneme pronunciation as the end point Display in correspondence with the note image of one note, and change the position of the end point of the preceding phoneme image on the pitch axis according to the pitch of the immediately preceding note.
  A display control method realized by a computer system.

Images