JP4858173B2 - Singing sound synthesizer and program - Google Patents

Description

  The present invention relates to a singing sound synthesizer that electrically synthesizes a singing voice, and more particularly to a technique for supporting input of music data representing notes, lyrics, facial expressions, and the like.

Stores lyric data and note data, reads out lyric data corresponding to reading out of note data, generates in advance sound data indicating phonemes corresponding to lyric data, and performs pronunciation according to the sound data Various kinds of singing sound synthesizers for singing lyrics have been proposed. This type of singing sound synthesizer is required to have a function of synthesizing natural human-like singing sounds. Observing human singing sounds includes changes in utterance strength, volume changes, breath components, etc. that are not directly expressed only by note data, and these are contributing to the human nature of singing Conceivable. Therefore, conventionally, studies have been made to give human-like facial expressions (that is, utterance change, volume change, breath component, etc.) to the singing sound synthesized by the singing sound synthesizer. For example, Non-Patent Document 1 provides a singing sound synthesizer with a user interface for prompting the user to input various parameters for giving a facial expression to the singing sound and for confirming the input contents individually. It has been proposed to add to the singing sound the sound effects and edits indicated by the parameters input via the face.
“Introduction to Cubase products”, [online], [Search January 5, 2007], Internet <URL: http://www.japan.steinberg.net/731.html>

  However, in the technique disclosed in Non-Patent Document 1, each parameter that gives a facial expression to the singing voice is displayed in a separate display area, so that a large display area is required when the number of parameters is large. Each parameter is expressed as time-series data sharing a time axis. However, since each parameter is displayed in a different area for each parameter, there is a problem that it is difficult to compare and refer to the parameters.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that facilitates input or editing of a plurality of time-series data sharing a time axis.

  In order to solve the above problems, the present invention provides musical score data in which note data indicating the pitches and pronunciation periods of notes constituting a musical piece and lyrics data indicating lyrics uttered by the notes are arranged in the order of the voices, A plurality of types of time-series data sharing a time axis with the score data, and a plurality of types of time-series data representing time changes of parameters indicating facial expressions given to the singing sound synthesized according to the score data According to the musical score data acquired by the acquisition means and the acquisition means, a musical score image showing a temporal change in pitch of the music and a temporal change in lyrics is displayed on the display device, while along the time axis of the musical score image. An image representing a graph showing a time change of each of the plurality of parameters is synthesized according to each time series data, and the time axis of each graph and the score image Display control means for displaying on the display device side by side with the musical score image in correspondence with the interval, and synthesize the singing sound of the music according to the musical score data acquired by the acquisition means, and the facial expression indicated by each parameter A singing sound synthesizing device having a singing sound synthesizing means for giving and outputting.

  In a more preferred aspect, the display control means of the singing sound synthesizing apparatus associates the time axis of each graph with a transparent process for making the background color transparent to the image of the graph showing the time change of each parameter. And the time axis is aligned with the time axis of the score image and displayed on the display device side by side with the score image. In another preferred embodiment, the display control means applies a color different for each parameter to the area between the graph showing the time change of each parameter and the time axis to correspond to the time axis of each graph. The colored regions that are overlapped and overlap each other are converted to a color corresponding to the overlapping state and displayed on the display device. In another preferable aspect, the singing sound synthesizer includes an operation unit and an update unit that updates the time-series data in accordance with an operation on the operation unit, and the display control unit includes the display device. When one of the plurality of graphs displayed on the screen is selected by the operation means, the transparent display is controlled so that the image of the graph can be seen in the foreground, and the update means controls the selected When an operation of rewriting a graph is performed by the operation means, the time series data corresponding to the graph is updated according to the rewriting operation.

  Further, in order to solve the above-described problem, the present invention provides a computer device for acquiring time-series data indicating a time change of a parameter value for each of a plurality of parameters that change with time along the same time axis. And a display control unit that combines the images representing the graphs showing the time changes of the plurality of parameters according to the time series data, and displays them on the display device in a superimposed manner while matching the time axes of the graphs. Program to be provided.

  According to the present invention, since a plurality of graphs indicated by a plurality of time-series data sharing a time axis are combined and displayed while sharing a time axis, the user can easily compare and reference a plurality of time-series data. There is an effect.

(A: 1st Embodiment)
FIG. 1 is a block diagram illustrating a configuration example of a singing sound synthesizer 10 according to an embodiment of the present invention. As shown in FIG. 1, the singing sound synthesizer 10 includes a control unit 110, an operation unit 120, a display unit 130, a voice output unit 140, a storage unit 150, and a bus 160 that mediates data exchange between each component. ing.

  The control unit 110 is, for example, a CPU (Central Processing Unit), and is a control center that performs operation control of each unit of the singing sound synthesizer 10. The operation unit 120 includes a keyboard having a plurality of operators such as a numeric keypad and a pointing device such as a mouse (all not shown). The operation unit 120 delivers data representing an operation performed on the operation element or the pointing device (for example, an identifier indicating the pressed operation element or data indicating the movement amount of the pointing device) to the control unit 110. As a result, the user's operation content on the operation unit 120 is transmitted to the control unit 110.

  The display unit 130 is, for example, a liquid crystal display and its drive circuit (both not shown), and displays an image represented by the image data delivered from the control unit 110. The audio output unit 140 includes a D / A converter, an amplifier, and a speaker (all not shown). The audio output unit 140 converts the audio data received from the control unit 110 into an audio signal by a D / A converter, amplifies the audio signal as appropriate by an amplifier, and gives it to a speaker to output a sound represented by the audio data.

  As shown in FIG. 1, the storage unit 150 includes a volatile storage unit 150a and a nonvolatile storage unit 150b. The volatile storage unit 150a is, for example, a RAM (Random Access Memory), and is used as a work area by the control unit 110 operating according to various software. On the other hand, the non-volatile storage unit 150b is a hard disk, for example, and stores various data and various programs.

  As an example of data stored in the nonvolatile storage unit 150b, data stored in the music database 152 and the phonological database 154 shown in FIG. The music database 152 shown in FIG. 1 is associated with a music identifier (for example, character string data indicating a music name) uniquely indicating a music, and musical score data used for synthesizing the song sound of the music, and the song A parameter indicating the facial expression given to the sound is stored. On the other hand, the phonological database 154 is a collection of phonological data that is referred to during singing synthesis. In this phoneme database 154, a collection of phoneme data is prepared in advance for each type of voice to be uttered, for example, male voice, female voice, or a specific singer.

  The musical score data includes note data indicating the pitch and pronunciation period (for example, utterance time and note length) of each note constituting the musical piece identified by the musical piece identifier associated with the musical score data, and the musical note. And lyrics data indicating lyrics to be uttered in accordance with. This score data is a chronological arrangement of note data and lyric data corresponding to individual notes in accordance with the utterance order from the start of the music. In the score data, note data and lyric data are Corresponds in units.

  On the other hand, parameters are classified into parameters indicating the utterance mode of notes and parameters relating to pitch bend. The parameters indicating the utterance mode of the note are “VEL” indicating the strength of the utterance of the note, “DYN” indicating the volume, “BRE” indicating the strength of the breath component, “BRI” indicating the tone of the voice, transparency of the voice There are seven types: “CLE” indicating a feeling, “GEN” indicating a change in gender and age, and “POR” indicating a smooth connection of musical notes. The parameters relating to pitch bend are two types, “PIT” indicating the presence or absence of pitch bend and “PBS” indicating the maximum width of the pitch bend. All these nine types of parameters are stored for each song in the song database 152 as an array of parameter values at the time on the time axis starting from the playback start point of the song (that is, time-series data sharing the time axis). Has been. In the present embodiment, the case where the above nine types are used as parameters for giving a human-like expression to the singing voice to be electrically synthesized will be described. However, any one of these nine types may be used. These nine types may be used by adding one or more other parameters. Hereinafter, a set of score data and each parameter stored in the music database 152 in association with a music identifier is also referred to as “music data”.

  As an example of the program stored in the non-volatile storage unit 150b, an OS program (not shown), a song editing program 156, and a song that causes the control unit 110 to realize a so-called operation system (hereinafter referred to as “OS”). An example is a synthesis program 158. The control unit 110 operating according to the OS program includes a function for controlling the operation of each unit of the singing sound synthesizer 10, a function for providing a GUI for prompting the user to perform various inputs, and others according to user instructions. The function of executing the program is provided. The song editing program 156 is a program for causing the control unit 110 to execute processing for displaying the song data input / output screen shown in FIG. 2 on the display unit 130 and prompting the user to browse and edit the song data. The music data input / output screen will be described in detail later. On the other hand, the singing synthesis program 158 is a process of synthesizing and outputting the singing sound of the music identified by the music identifier specified by the user with the voice quality and expression specified by the user, and the music data and phoneme corresponding to the music. This is a program to be executed by the control unit 110 according to the stored contents of the database. As described above, the process performed by the control unit 110 operating according to the OS program according to the song synthesis program 158 is the same as a general song synthesis process. For this reason, below, it demonstrates focusing on the process which the control part 110 which is operate | moving according to the music edit program 156 performs.

  When an operation for instructing execution of the song editing program 156 is performed on the operation unit 120, the control unit 110 operating according to the OS program receives data indicating the operation content from the operation unit 120, In accordance with the instruction indicated by the data, the music editing program 156 is read from the nonvolatile storage unit 150b to the volatile storage unit 150a, and its execution is started. As shown in FIG. 3, the control unit 110 operating according to the song editing program functions as software modules of the data access unit 310, the display control unit 320, and the operation content determination unit 330.

  The data access means 310 has a function of acquiring music data from the music database 152 and updating the stored contents of the music database 152. More specifically, the data access unit 310 reads out music data corresponding to the music identifier input via the operation unit 120 from the music database 152, and associates the music data with the music identifier in the volatile storage unit 150a. Write to the area. The music data written in the predetermined area of the volatile storage unit 150a is used by the display control unit 320 when the music data input / output screen (see FIG. 2) is displayed on the display unit 130. The music data stored in the predetermined area in the volatile storage unit 150a is appropriately updated according to the editing operation performed by the user on the music data input / output screen. When a registration instruction is input via the operation unit 120, the data access unit 310 overwrites and updates the corresponding music data in the music database 152 with the music data stored in the predetermined area.

  The display control unit 320 displays the music data input / output screen shown in FIG. 2 on the display unit 130 according to the music data written in the predetermined area in the volatile storage unit 150a by the data access unit 310. As shown in FIG. 2, the music data input / output screen is divided into three input / output areas: a score data input / output area 210, a parameter selection area 220, and a parameter input / output area 230.

  The score data input / output area 210 is used to store the singing sound of the song identified by the song identifier according to the score data included in the song data stored in the predetermined area in the volatile storage unit 150a in association with the song identifier. This is an input / output area for displaying an image showing a temporal change in pitch (hereinafter also referred to as “pitch”) and lyrics. In the score data input / output area 210, a bar-like image having a length corresponding to the length of the note indicated by the note data is displayed on the coordinate plane in which the vertical axis indicates the pitch and the horizontal axis indicates the time. A musical score image arranged at a coordinate position corresponding to the pitch and utterance time indicated by the data is displayed by the display control means 320. As shown in FIG. 2, this score image is also called a piano roll image because images simulating a piano keyboard are arranged along the vertical axis as a pitch scale. As is clear from FIG. 2, in the musical score image according to the present embodiment, a character string indicating the lyrics indicated by the lyrics data corresponding to the note data in the vicinity of the bar-shaped image corresponding to each note data. An image is placed. The user who visually recognizes the score image displayed in the score data input / output area 210 can intuitively grasp the time change of the pitch of the music and the lyrics from the arrangement of the bar-shaped images in the score image. In the present embodiment, a case where a piano roll image is used as a musical score image representing a time change of the pitch and lyrics of music is described. For example, a musical score image in which musical notes are recorded on a staff according to each musical note data. Of course, it is also possible to use.

  In the parameter selection area 220, an image for instructing the user which of the nine types of parameters described above is to be displayed or edited is displayed. As shown in FIG. 2, the image displayed in the parameter selection area 220 includes a check box (hereinafter referred to as a display instruction box) for instructing the user of a display target from the nine types of parameters described above, A check box (hereinafter referred to as an edit instruction box) is included for allowing the user to instruct the user to edit one of the parameters selected by the user. By appropriately operating the operation unit 120 and clicking each check box, the user can select a parameter to be displayed or select one to be edited from the displayed parameters. Which check box is designated or canceled by the user is determined by the operation content determination unit 330 by analyzing data delivered from the operation unit 120, and data indicating the determination result is delivered to the display control unit 320. It is. On the other hand, the display control unit 320 displays, in the parameter input / output area 230, a graph image showing a time change of a parameter designated as a display target every time data indicating the operation content of the user is received from the operation content determination unit 330. . Note that the display control unit 320 displays a background image such as a white color in the parameter input / output area 230 in a situation where a parameter to be displayed has not yet been selected.

  More specifically, when the display control unit 320 receives data indicating that the display instruction box in the parameter selection area 220 has been selected from the operation content determination unit 330, the time change of the parameter corresponding to the display instruction box Is generated in the transmission mode, which represents a graph indicating the same as the score image on the same time axis. Here, the transparent mode is a mode in which image data is generated by performing a transparent process (that is, a process for making the background portion of the graph transparent). Then, the display control unit 320 displays the image represented by the image data generated in the transmission mode so as to be superimposed on the parameter input / output area 230. For example, an operation of selecting display designation boxes in this order for three types of parameters, “VEL” indicating the strength of the utterance of a note, “DYN” indicating the volume, and “BRE” indicating the strength of the breath component, was performed. In this case, as shown in FIG. 4A, the display control unit 320 superimposes a graph showing the time change of each parameter on the base image in the selection order and displays the graph in the parameter input / output area 230. Here, when the three types of parameters are changing with time as shown in FIGS. 5A, 5B and 5C, the music data input / output screen shown in FIG. Will be displayed. In this example, as is clear from FIG. 6, the image of each graph displayed superimposed on the parameter input / output area 230 has been subjected to transparency processing. It is possible to grasp the time change of all at once.

  On the other hand, when the display control unit 320 receives data indicating that the edit instruction box has been selected from the operation content determination unit 330, the display control unit 320 draws a line for the graph on the image representing the parameter graph corresponding to the edit instruction box. Move to the foreground after processing to change to a dotted line. The reason why the line type of the parameter specified as the editing target is different from other graphs is to make the editing target graph clear to the user, but the stroke color of the editing target graph is different from the other graphs. It is also possible to make the user clearly grasp the editing target by making it different from the above.

  For example, in a situation where the music data input / output screen shown in FIG. 6 is displayed on the display unit 130, when the edit designation box for “VEL” indicating the strength of the utterance of a note is selected, the display control means 320 After the processing of changing the type of the drawing line of the graph is performed on the image of the graph showing the time change of the parameter (see FIG. 5A), as shown in FIG. Move to the front. As a result, the music data input / output screen shown in FIG. 7 is displayed on the display unit 130. In this example, since the graph displayed in the forefront of the parameter input / output area 230 is drawn with a line of a different type from the other graphs, the graph to be edited can be clearly understood by the user. Further, since each graph image displayed in the parameter input / output area 230 of the song data input / output screen shown in FIG. It becomes easy to update the time change.

  As described above, according to the singing sound synthesizing apparatus 10 according to the present embodiment, each of the time-varying parameters of the plurality of parameters selected by the user is displayed together with the score image indicating the time-varying pitch of the singing sound of the music. An image synthesized by superimposing the graphs is displayed along the time axis of the score image. For this reason, it is not necessary to use an input / output area for each parameter, and the drawing area of the display unit 130 is not used wastefully, and the time change of each parameter represented by a plurality of types of time-series data sharing a time axis can be obtained. There is an effect that comparison / reference and editing are facilitated. In the present embodiment, the case where the time series data display function characteristic of the singing sound synthesizer according to the present invention is realized by a software module has been described, but it is needless to say that it may be realized by a hardware module. .

(B: Other embodiments)
Although one embodiment of the present invention has been described above, it goes without saying that modifications described below may be added to such an embodiment.
(1) In the above-described embodiment, a case has been described in which a single computer apparatus performs a song data editing process and a song synthesis process for performing song synthesis according to song data. However, the music data editing process and the song composition process may be executed by separate computer devices.

(2) In the above-described embodiment, a case has been described in which a graph indicating a time change of each of a plurality of parameters designated for display by a user is subjected to a transparent process and then superimposed and displayed. In order to make it easy to grasp, the type of line used for graph drawing may be different for each parameter, and of course, the color of the line may be different. Further, instead of indicating each parameter by the color or type of a line for drawing a graph, a different color for each parameter or a color according to the selection order may be applied to a region between the graph curve and the time axis. Specifically, the first selected parameter is red, the second selected parameter is yellow, the third selected parameter is blue, and the distance between the graph curve and the time axis What is necessary is just to color an area | region. When these colored graph images are subjected to transparency processing and superimposed, the colored areas are converted to colored and transparent, and the colored areas that overlap each other are converted to colors according to the degree of overlap. What is necessary is just to give the process to do. In the above example, if the yellow colored area overlaps the red colored area, the overlapping part may be converted to yellowish orange, and the yellow colored area When the area colored in blue overlaps, the overlapping portion may be converted to bluish green. By performing such conversion processing, it is possible to make the user understand the overlapping order of the parameters by the hue of the overlapping portion. An existing image processing algorithm can be used for determining the color of the overlapped portion and for converting the color.

(3) In the above-described embodiment, the case has been described in which the score image indicating the time change of the pitch of the music from the beginning of the music to the end of the music and the image indicating the time change of the various parameters are collectively displayed on the display unit 130. However, when a song editing program is executed on a computer device that has a narrow drawing area compared to a stationary personal computer such as a notebook personal computer or PDA (Personal Digital Assistants), for example, the entire score image for a single song is batched. In some cases, display cannot be performed or display cannot be performed unless the image is reduced so as not to be visually recognized. Therefore, if there is no drawing area that is large enough to display the entire score image for one song, a scroll bar and scroll buttons are provided as in the case of a general word processor screen or spreadsheet screen. The score image may be divided and displayed for each time range and for each predetermined pitch range. However, when displaying a score image for each predetermined time range and switching the time range with a scroll button etc., the time range of the graph showing the time change of the parameter is also displayed in synchronization with the time range of the score image. It is necessary to switch.

(4) In the above-described embodiment, the process of combining the graph representing the time change of each parameter represented by each of a plurality of types of time-series data sharing the time axis into one image and causing the display unit 130 to display the graph. The case where the time series data display control program to be executed by the control unit 110 is installed in the singing sound synthesizer 10 in advance has been described. However, the time-series data display control program may be distributed by being written on a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) or a DVD (Digital Versatile Disk), or the Internet. The time-series data display control program may be distributed by downloading via a telecommunication line. By installing the time-series data display control program distributed in this way on a general computer device, the same function as the parameter display control module described above can be given to the computer device.

It is a block diagram which shows the structural example of the singing sound synthesizer 10 which concerns on one Embodiment of this invention. It is a figure which shows the screen layout of the music data input / output screen displayed on the display part 130 of the singing sound synthesizer. It is a figure which shows an example of the software module implement | achieved when the control part 110 of the singing sound synthesizer 10 operate | moves according to a music edit program. It is a figure for demonstrating the image composition process which the control part 110 performs according to a music edit program. It is an example of the graph which shows the time change of a parameter. It is a figure which shows an example of the music data input / output screen displayed on the display part 130 of the singing sound synthesizer. It is a figure which shows an example of the music data input / output screen displayed on the display part 130 of the singing sound synthesizer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Singing sound synthesizer, 110 ... Control part, 120 ... Operation part, 130 ... Display part, 140 ... Audio | voice output part, 150 ... Memory | storage part, 150a ... Volatile memory part, 150b ... Nonvolatile memory part, 160 ... Bus .

Claims (4)

Musical score data in which musical note data indicating the pitches and pronunciation periods of the musical notes constituting the musical piece and lyrics data indicating the lyrics uttered by the musical notes are arranged in the order of voice production, and a plurality of types sharing the time axis with the musical score data Time series data, and a plurality of types of time series data representing a time change of a parameter indicating a facial expression given to a singing sound synthesized according to the musical score data;
In accordance with the musical score data acquired by the acquisition means, the musical score image showing the time change of the pitch of the music and the time change of the lyrics is displayed on the display device, while the plurality of parameters along the time axis of the musical score image An image representing a graph showing each time change is synthesized according to each time series data, and the time axis of each graph and the time axis of the score image are associated with each other and displayed on the display device along with the score image. Display control means,
Synthesizer singing sound of the music according to the musical score data acquired by the acquiring means, and singing sound synthesizing means for providing and outputting a facial expression indicated by each parameter,
The display control means includes
In each of the graph images showing the time change of each parameter, after coloring the area between the graph curve showing the time change of the parameter and the time axis with a different color for each parameter , the time axis of each graph is changed. Colored areas that are overlapped and overlapped with each other are colors obtained by mixing the colors colored in the respective areas, and are converted into colors having a tint in the foreground area and displayed on the display device. A singing sound synthesizer characterized in that The singing voice synthesizing apparatus according to claim 1, wherein the display control unit draws a graph displayed on the foreground with a line of a type different from other graphs. An operation means, and an update means for updating the time-series data in accordance with an operation on the operation means,
The display control means, when one of a plurality of graphs displayed on the display device is selected by the operation means, performs display control so that an image of the graph can be seen in the foreground,
It said updating means, according to claim 1 or 2 operation to rewrite the selected graph when made by said operating means, and updates the time-series data corresponding to the graph in accordance with the rewriting operation The singing sound synthesizer described in 1. Computer
Musical score data in which musical note data indicating the pitches and pronunciation periods of the musical notes constituting the musical piece and lyrics data indicating the lyrics uttered by the musical notes are arranged in the order of voice production, and a plurality of types sharing the time axis with the musical score data Time series data, and a plurality of types of time series data representing a time change of a parameter indicating a facial expression given to a singing sound synthesized according to the musical score data;
In accordance with the musical score data acquired by the acquisition means, the musical score image showing the time change of the pitch of the music and the time change of the lyrics is displayed on the display device, while the plurality of parameters along the time axis of the musical score image An image representing a graph showing each time change is synthesized according to each time series data, and the time axis of each graph and the time axis of the score image are associated with each other and displayed on the display device along with the score image. In each of the graph images showing the time change of each parameter, the region between the graph curve representing the time change of the parameter and the time axis is colored with a different color for each parameter . superimposed in correspondence with the time axis of each graph, for colored region overlap each other a color obtained by mixing the color colored to each area, the outermost Display control means for displaying on the display device by converting the color of tint color of the surface area,
A program that synthesizes the singing sound of the music according to the musical score data acquired by the acquiring means, and that functions as a singing sound synthesizing means that outputs the facial expression indicated by each parameter.

Images