JP2024054051A - System and method for training an acoustic model - Google Patents

Abstract

[Problem] To make it possible to easily perform various training by making it possible to select data to be used for training an acoustic model from a plurality of training data. [Solution] An acoustic model training system includes a first device connectable to a network and used by a first user, and a server connectable to the network. The first device uploads a plurality of sound waveforms to the server under the control of the first user, selects one or more sound waveforms as a first waveform set from the plurality of sound waveforms that have already been uploaded or will be uploaded, and transmits a first execution instruction for a first training job for an acoustic model that generates acoustic features to the server. Based on the first execution instruction from the first device, the server starts execution of the first training job using the selected first waveform set, and provides the first device with a trained acoustic model trained by the first training job. [Selected Figure] Figure 4

Description

One embodiment of the present invention relates to a system and method for training an acoustic model.

Sound synthesis technology is known that synthesizes the vocal sounds of a specific singer and the sounds played on a specific instrument. In particular, in voice synthesis technology that uses machine learning (e.g., Patent Documents 1 and 2), a fully trained acoustic model is required to output a synthetic voice with natural pronunciation from the specific voice and performance sounds based on the sheet music data and audio data input by the user.

JP 2020-076843 A International Publication No. 2022/080395

However, to fully train an acoustic model, it was necessary to label a huge amount of speech and performance sounds with linguistic features, which required a huge amount of time and money. As a result, only well-funded companies were able to train acoustic models, and the types of acoustic models available were limited.

One of the objectives of one embodiment of the present invention is to make it easy to perform various types of training by allowing the selection of data to be used for training an acoustic model from multiple training data.

An acoustic model training system according to an embodiment of the present invention includes a first device connectable to a network and used by a first user, and a server connectable to the network. The first device, under the control of the first user, uploads a plurality of sound waveforms to the server, selects one or more sound waveforms as a first waveform set from the plurality of sound waveforms that have already been uploaded or will be uploaded, and transmits a first execution instruction for a first training job for an acoustic model that generates acoustic features to the server. Based on the first execution instruction from the first device, the server starts execution of the first training job using the selected first waveform set, and provides the first device with a trained acoustic model trained by the first training job.

An acoustic model training method according to one embodiment of the present invention includes providing a first user with an interface that allows the first user to select one or more sound waveforms from a plurality of pre-stored sound waveforms for executing a first training job for an acoustic model that generates acoustic features, using one or more computers.

According to one embodiment of the present invention, by making it possible to select data to be used for training an acoustic model from multiple training data, it becomes possible to easily perform various types of training.

1 is a diagram showing the overall configuration of an acoustic model training system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of a server in one embodiment of the present invention. FIG. 2 is a block diagram showing the concept of an acoustic model in one embodiment of the present invention. FIG. 2 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in an embodiment of the present invention. FIG. 2 is a diagram showing an example of a GUI in a method for training an acoustic model in one embodiment of the present invention. FIG. 2 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in an embodiment of the present invention. FIG. 13 is a diagram showing an example of a GUI related to disclosure of acoustic model information and a preview request in one embodiment of the present invention. FIG. 2 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in an embodiment of the present invention. FIG. 13 is a diagram showing an example of a GUI for setting public information during training of an acoustic model in one embodiment of the present invention. 4 is a flow chart illustrating a method for training an acoustic model in accordance with an embodiment of the present invention. FIG. 2 is a sequence diagram showing a method for recording sound waveforms used for training an acoustic model in one embodiment of the present invention. FIG. 2 is a diagram showing a data structure managed by a server in one embodiment of the present invention. FIG. 2 illustrates data sent to a server in training an acoustic model in one embodiment of the present invention. 4 is a flow chart illustrating a method for training an acoustic model in accordance with an embodiment of the present invention. 1 is a flow chart illustrating a method for recommending songs suitable for training an acoustic model in accordance with an embodiment of the present invention.

The following describes in detail an acoustic model training system and method according to one embodiment of the present invention with reference to the drawings. The embodiments described below are examples of ways of implementing the present invention, and the present invention is not to be interpreted as being limited to these embodiments. In the drawings referred to in this embodiment, identical parts or parts having similar functions are given the same or similar symbols (symbols consisting of only a number followed by A, B, etc.), and repeated explanations of them may be omitted.

In the following embodiments, "musical score data" is data that includes information about the pitch and intensity of notes, information about the phonemes of notes, information about the duration of note pronunciation, and information about performance symbols. For example, musical score data is data that indicates at least one of the musical score and lyrics of a piece of music. Music score data may be data that indicates the time sequence of notes that make up the piece of music, or data that indicates the time sequence of language that makes up the piece of music.

A "sound waveform" is waveform data of a voice, and the sound source that produces the voice is identified by a sound source ID. For example, a sound waveform is at least one of waveform data of singing and waveform data of musical instrument sounds. For example, a sound waveform includes waveform data of a singer's singing voice and musical instrument sounds captured via an input device such as a microphone. The sound source ID identifies the timbre of the singer's singing or the timbre of the musical instrument sounds. Among the sound waveforms, a sound waveform that is input to generate a synthetic sound waveform using an acoustic model is called a "synthesis sound waveform," and a sound waveform used to train the acoustic model is called a "training sound waveform." When there is no need to distinguish between synthesis sound waveforms and training sound waveforms, they are collectively simply called "sound waveforms."

The "acoustic model" has an input of the score feature of the score data and an input of the acoustic feature of the sound waveform. For example, an acoustic model having a score encoder 111, an acoustic encoder 121, a switching unit 131, and an acoustic decoder 133 described in International Publication No. 2022/080395 is used as the acoustic model. This acoustic model has a function of generating acoustic features of a target sound waveform having a timbre indicated by the sound source ID by processing the score feature of the input score data or the acoustic feature of the sound waveform and the sound source ID, and is a voice synthesis model used by a voice synthesis program for generating a new synthetic sound waveform. The voice synthesis program supplies the score feature generated from the score data of a certain song and the sound source ID to the acoustic model, thereby obtaining the acoustic feature of the song with the timbre indicated by the sound source ID, and converting the acoustic feature into a sound waveform. Alternatively, the voice synthesis program supplies acoustic features generated from the sound waveform of a certain piece of music and a sound source ID to an acoustic model, thereby obtaining new acoustic features for that piece of music in the timbre indicated by the sound source ID, and converting the new acoustic features into a sound waveform. A predetermined number of sound source IDs are prepared for each acoustic model. In other words, each acoustic model selectively generates acoustic features for the timbre indicated by the sound source ID from among a predetermined number of timbres.

An acoustic model is a generative model of a given architecture that uses machine learning, for example, a convolutional neural network (CNN) or a recurrent neural network (RNN). Acoustic features represent the pronunciation characteristics in the frequency spectrum of the waveform of natural sounds or synthetic sounds, and similar acoustic features mean that the timbre of the singing voice or performance sound and its changes over time are similar.

In training the acoustic model, the variables of the acoustic model are changed so that the acoustic model generates acoustic features similar to those of the referenced sound waveform. For example, the training program P2 described in International Publication No. 2022/080395, the score data D1 (training score data), and the learning sound data D2 (training sound waveform) are used for training. The variables of the acoustic model (score encoder, sound encoder, and sound decoder) are changed so that acoustic features of a synthetic sound of a plurality of timbres corresponding to a plurality of sound source IDs can be generated by basic training using waveforms of a plurality of voices corresponding to a plurality of sound source IDs. Furthermore, by supplementarily training the trained acoustic model using a sound waveform of another timbre corresponding to a new (unused) sound source ID, the acoustic model can generate acoustic features of the timbre indicated by the new sound source ID. Specifically, by further performing supplementary training on an acoustic model already trained with the sound waveforms of the voices of XXX (multiple people) using a new sound source ID and the sound waveforms of the voice of YYY (one person), the variables of the acoustic model (at least the acoustic decoder) are changed so that the acoustic model becomes one that can generate acoustic features of YYY's voice. The unit of training of the acoustic model corresponding to the new sound source ID as described above is called a "training job." In other words, a training job refers to a series of training processes executed by a training program.

A "program" refers to an instruction or group of instructions executed by a processor in a computer equipped with a processor and memory. A "computer" is a general term referring to the entity that executes a program. For example, when a program is executed by a server (or a client), the "computer" refers to the server (or the client). Also, when a "program" is executed by distributed processing between a server and a client, the "computer" includes both the server and the client. In this case, the "program" includes "a program executed by a server" and "a program executed by a client." Similarly, when a "program" is processed in a distributed manner among multiple servers, the "computer" includes the multiple servers, and the "program" includes each program executed by each server.

[1. First embodiment]
[1-1. Overall system configuration]
Fig. 1 is a diagram showing the overall configuration of an acoustic model training system in one embodiment of the present invention. As shown in Fig. 1, the acoustic model training system 10 includes a server 100, a communication terminal 200 (TM1), and a communication terminal 300 (TM2). The server 100 and the communication terminals 200, 300 can each be connected to a network 400. The communication terminals 200 and 300 can each communicate with the server 100 via the network 400. The communication terminal 200 may be referred to as a "first device". A user who uses the communication terminal 200 may be referred to as a "first user".

In this embodiment, the server 100 is a computer that functions as a speech synthesizer and trains an acoustic model. The server 100 includes a storage 110. FIG. 1 illustrates a configuration in which the storage 110 is directly connected to the server 100, but the configuration is not limited to this. For example, the storage 110 may be connected to the network 400 directly or via another computer, and data may be transmitted and received between the server 100 and the storage 110 via the network 400.

The communication terminal 200 is a terminal that selects a training sound waveform for training an acoustic model and transmits an instruction to the server 100 to execute the training. The communication terminal 300 is a terminal different from the communication terminal 200 and is a terminal that can access the server 100. As will be described in detail later, the communication terminal 300 is a terminal for viewing or listening to public information related to the acoustic model under training. The communication terminals 200 and 300 include, for example, mobile communication terminals such as smartphones or tablet terminals, or stationary communication terminals such as desktop personal computers.

Network 400 may be the Internet provided by a typical World Wide Web (WWW) service, a Wide Area Network (WAN), or a Local Area Network (LAN) such as an in-house LAN.

[1-2. Configuration of the server used for speech synthesis]
Fig. 2 is a block diagram showing the configuration of a server according to an embodiment of the present invention. As shown in Fig. 2, the server 100 includes a control unit 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a user interface (UI) 104, a communication interface 105, and a storage 110. The speech synthesis technology of the present embodiment is realized by the cooperation of the functional units of the server 100.

The control unit 101 includes a central processing unit (CPU), an image processing unit (GPU), and storage devices such as registers and memory connected to the CPU and GPU. The control unit 101 executes programs temporarily stored in the memory using the CPU and GPU, and realizes each function provided in the server 100. Specifically, the control unit 101 performs calculations in response to various request signals from the communication terminal 200, and provides content data to the communication terminals 200 and 300.

RAM 102 temporarily stores control programs required for computational processing, acoustic models (consisting of architecture and variables), content data, and the like. RAM 102 is also used, for example, as a data buffer, and temporarily holds various data received from external devices such as communication terminal 200 until the data is stored in storage 110. For example, general-purpose memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory) may be used as RAM 12.

The ROM 103 stores various programs, various acoustic models, parameters, etc. for implementing the functions of the server 100. The programs, acoustic models, parameters, etc. stored in the ROM 103 are read by the control unit 101 as necessary and executed or used.

Under the control of the control unit 101, the user interface 104 displays various display images such as a graphical user interface (GUI) on its display device and accepts input from the user of the server 100.

The communication interface 105 is an interface that connects to the network 400 under the control of the control unit 101, and transmits and receives information between the communication interface 105 and other communication devices, such as the communication terminals 200 and 300, that are connected to the network 400.

The storage 110 is a recording device (recording medium) capable of permanently retaining and rewriting information, such as a non-volatile memory or a hard disk drive. The storage 110 stores information such as a program, an acoustic model, and parameters required for executing the program. As shown in FIG. 2, the storage 110 stores, for example, a voice synthesis program 111, a training job 112, musical score data 113, and a sound waveform 114. These programs and data can be those related to general voice synthesis, and for example, the voice synthesis program P1, training program P2, musical score data D1, and acoustic data D2 described in International Publication No. WO 2022/080395 may be used.

As described above, the voice synthesis program 111 is a program for generating a synthetic voice waveform from musical score data and sound waveforms. When the control unit 101 executes the voice synthesis program 111, the control unit 101 generates a synthetic voice waveform using the acoustic model 120. The synthetic voice waveform corresponds to the acoustic data D3 described in WO 2022/080395. The training program for the acoustic model 120 executed by the control unit 101 in the training job 112 is, for example, a program for training the encoder and acoustic decoder described in WO 2022/080395. The musical score data is data that specifies a musical piece. The sound waveform is waveform data of a voice or a performance sound, for example, waveform data indicating the singing voice of a singer or the performance sound of an instrument.

[1-3. Functional configuration of server used for speech synthesis]
FIG. 3 is a block diagram showing the concept of an acoustic model in one embodiment of the present invention. As described above, the acoustic model 120 is a machine learning model used in the voice synthesis technology executed by the control unit 101 in FIG. 2 when the control unit 101 reads and executes the voice synthesis program 111. The acoustic model 120 generates an acoustic feature. The control unit 101 inputs the score feature 123 of the score data 113 of a desired piece of music or the acoustic feature 124 of the sound waveform 114 as an input signal to the acoustic model 120. The acoustic model 120 processes the sound source ID and the score feature 123 to generate an acoustic feature 129 of the synthetic sound of the piece of music. The control unit 101 synthesizes and outputs a synthetic sound waveform 130 of the piece of music sung by a singer specified by the sound source ID or played on an instrument based on the acoustic feature 129. Alternatively, the acoustic model 120 processes the sound source ID and the acoustic feature 124 to generate the acoustic feature 129 of the synthetic sound of the piece of music. Based on the acoustic feature quantity 129, the control unit 101 synthesizes and outputs a synthetic sound waveform 130 by converting the sound waveform of the song into the tone color of the singer's singing voice or the performance sound of an instrument specified by the sound source ID.

The acoustic model 120 is a generative model that uses machine learning, and is trained by the control unit 101 that is executing a training program (i.e., that is executing the training job 112). The control unit 101 trains the acoustic model 120 using a (new) sound source ID and a training sound waveform, and determines the variables of the acoustic model 120 (at least the acoustic decoder). Specifically, the control unit 101 generates training acoustic features from the training sound waveform, and when a new sound source ID and training acoustic features are input to the acoustic model 120, gradually and repeatedly changes the above variables so that the acoustic features that generate the synthetic sound waveform 130 approach the training acoustic features when the new sound source ID and training acoustic features are input to the acoustic model 120. The training sound waveform may be uploaded (transmitted) from the communication terminal 200 or the communication terminal 300 to the server 100, for example, and stored in the storage 110 as user data, or may be stored in the storage 110 in advance by the administrator of the server 100 as reference data. In the following description, storing in the storage 110 may be referred to as storing in the server 100.

[1-4. Voice synthesis method]
FIG. 4 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in an embodiment of the present invention. In the method for training an acoustic model shown in FIG. 4, an example is shown in which the communication terminal 200 uploads a training sound waveform to the server 100. However, as described above, the training sound waveform may be stored in advance in the server 100 by other methods. The training job in the sequence shown in FIG. 4 may be referred to as a "first training job". Each step of the process TM1 on the communication terminal 200 side and each step of the process Server on the server 100 side are actually executed by the control unit of the communication terminal 200 and the control unit 101 of the server 100, respectively, but here, for simplicity of explanation, the communication terminal 200 and the server 100 are expressed as the executing subjects of each step. Unless otherwise specified, the following explanation of the sequence diagram and the explanation of the flowchart are the same.

As shown in FIG. 4, first, the communication terminal 200 (first device) uploads (transmits) one or more training sound waveforms to the server 100 based on instructions from the first user who has logged in to the first user's account on the server 100 (step S401). The server 100 stores the training sound waveforms transmitted in S401 in the first user's memory area (step S411). Either one or multiple sound waveforms may be uploaded to the server 100, and multiple sound waveforms may be stored in multiple folders in the first user's memory area. The above steps S401 and S411 are steps related to preparation for executing the following training job.

Next, the steps for executing a training job are described below. The communication terminal 200 requests the server 100 to execute a training job (step S402). In response to the request of S402, the server 100 provides the communication terminal 200 with a graphical user interface (GUI) for selecting a sound waveform to be used for the training job from among the sound waveforms previously stored (and sound waveforms to be stored) (step S412).

The communication terminal 200 displays the GUI provided in S412 on its display, and the first user uses the GUI to select one or more training sound waveforms from the multiple sound waveforms uploaded to the storage area (or a desired folder) as a waveform set 149 (see FIG. 5) (step S403). After the waveform set 149 (training sound waveforms) is selected in S403, the communication terminal 200 instructs the first user to start executing a training job in response to an instruction from the first user (step S404).

Based on the instruction from the communication terminal 200 (first device) in S404, the server 100 starts executing the training job using the selected waveform set 149 (step S413). In other words, in S413, the training job is executed based on the instruction of the first user via the GUI provided in S412.

For training, not all of the waveforms in the selected waveform set 149 are used, but a preprocessed waveform set that includes only useful sections excluding silent sections and noise sections is used. In addition, an acoustic model 120 with an untrained acoustic decoder may be used as the acoustic model 120 to be trained (base acoustic model), but the time and cost required for the training job can be reduced by selecting and using an acoustic model 120 from multiple basic trained acoustic models 120 that includes an acoustic decoder that has learned to generate acoustic features similar to those of the waveforms in the waveform set 149. Whichever acoustic model 120 is selected, the score encoder and acoustic encoder are used that have been basic trained.

The base acoustic model may be determined by the server 100 based on the waveform set 149 selected by the first user. Alternatively, the first user may select one of a plurality of trained acoustic models as the base acoustic model and include designation data indicating the base acoustic model in the first execution instruction. A new, unused sound source ID is used as the sound source ID (e.g., singer ID, instrument ID, etc.) to be supplied to the acoustic decoder. Here, the user does not necessarily need to know which sound source ID has been used as the new sound source ID. However, when synthesizing speech using the trained model, the new sound source ID is automatically used.

In the training job, a unit training is repeated in which some short waveforms are extracted little by little from the preprocessed waveform set, and the extracted short waveforms are used to train the acoustic model (at least the acoustic decoder). In the unit training, the new sound source ID and the acoustic features of the short waveforms are input to the acoustic model 120, and the variables of the acoustic model are adjusted accordingly so that the difference between the acoustic features output by the acoustic model 120 and the input acoustic features becomes small. For example, the backpropagation method is used to adjust the variables. By repeating the unit training, once the training using the preprocessed waveform set is completed, the quality of the acoustic features generated by the acoustic model 120 is evaluated, and if the quality does not reach a predetermined standard, the preprocessed waveform set is used to train the acoustic model again. If the quality of the acoustic features generated by the acoustic model 120 reaches a predetermined standard, the training job is completed, and the acoustic model 120 at that point becomes the trained acoustic model 120.

When the training job is completed in S413, the trained acoustic model 120 is established (step S414). This trained acoustic model 120 may be referred to as the "first acoustic model." The server 100 notifies the communication terminal 200 that the trained acoustic model 120 has been established (step S415). The above steps S403 to S415 are the training job for the acoustic model 120.

After the notification in S415, in response to an instruction from the first user, the communication terminal 200 transmits a voice synthesis instruction including the score data of the desired piece of music to the server 100 (step S405). In response, the server 100 executes a voice synthesis program and performs voice synthesis using the trained acoustic model 120 completed in S414 based on the score data (step S416). The synthetic sound waveform 130 generated in S416 is transmitted to the communication terminal 200 (step S417). In this voice synthesis, the new sound source ID is used.

By combining S416 and S417, it can be said that the trained acoustic model 120 (voice synthesis function) trained by the training job is provided to the communication terminal 200 (first device) or the first user. The execution of the voice synthesis program in step S416 may be performed by the communication terminal 200 instead of the server 100. In this case, the server 100 transmits the trained acoustic model 120 to the communication terminal 200, and the communication terminal 200 uses the received trained acoustic model 120 to execute voice synthesis processing based on the sheet music data of the desired song with the new sound source ID, and obtains the synthetic voice waveform 130.

In this embodiment, the training sound waveform is uploaded in S401 before the execution of the training job is requested in S402, but this configuration is not limited to this. For example, the training sound waveform may be uploaded after the execution of the training job is instructed in S404. In this case, in S403, one or more sound waveforms are selected as waveform set 149 from a plurality of sound waveforms (including sound waveforms that have not been uploaded) stored in the communication terminal 200, and the sound waveforms that have not been uploaded from the selected sound waveforms may be uploaded in response to the instruction to execute the training job.

[1-5. GUI 140]
An example of the GUI provided in S412 will now be described. Fig. 5 is a diagram showing an example of a first GUI in the acoustic model training method according to an embodiment of the present invention. The GUI 140 shown in Fig. 5 is a GUI for The training sound waveform is displayed on a display included in the user interface of the terminal 200. As shown in FIG. 5, the GUI 140 displays sound waveform A, sound waveform B, ..., sound waveform Z ( For example, the training waveforms uploaded to a specific folder are displayed. Next to each of the training waveforms are check boxes 141, 142, ..., 143. The sound waveform A, sound waveform B, ..., sound waveform Z displayed as shape candidates are, for example, sound waveforms related to the singing voice of the same person, and each may be a different song or singing style. The forms may be different sounds played on the same instrument.

In other words, in S412, the server 100 provides the communication terminal 200 with a GUI that allows the first user to select one or more sound waveforms as a waveform set 149 from a plurality of pre-stored sound waveforms (and sound waveforms to be stored) for executing a training job for the acoustic model 120.

In S403 above, the first user of the communication terminal 200 checks the check boxes 141, 142, ..., 143 shown in Figure 5 to select training sound waveforms. Figure 5 shows an example in which check boxes 141 and 142 are checked as training sound waveforms, and sound waveform A and sound waveform B are selected as waveform set 149. One or more waveforms may be selected as waveform set 149.

In the above S404, when the execute button 144 is pressed while the check boxes 141 and 142 are selected, the communication terminal 200 executes the instruction for the training job in S404. In response to the instruction for the training job, the server 100 starts training the acoustic model 120 using the waveform set 149 consisting of sound waveform A and sound waveform B. Pressing the execute button 144 includes clicking or tapping the execute button 144.

As described above, the acoustic model training system 10 according to this embodiment selects one or more sound waveforms from a plurality of sound waveforms (and sound waveforms to be stored) stored in advance in the storage 110, and executes a training job for the acoustic model 120 using the selected sound waveforms as training sound waveforms. With the above configuration, the first user of the communication terminal 200 obtains a desired acoustic model 120 by training an untrained or trained acoustic model 120. Note that the sound waveforms may be uploaded to the server 100 after the selection of the waveform set 149 or the instruction to execute a training job. In other words, the sound waveforms used in the training job may be uploaded from the communication terminal 200 to the server 100 at any time before the training job is started. Also, if the acoustic decoder is auxiliary training of a trained acoustic model, the trained acoustic model 120 can be obtained in a short time and at a low cost compared to the conventional acoustic model 120.

[2. Second embodiment]
An acoustic model training system 10A according to the second embodiment will be described with reference to Figures 6 and 7. The overall configuration of the acoustic model training system 10A and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore their description will be omitted. In the following description, description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to Figures 1 to 5, and the alphabet "A" will be added after the reference numerals shown in these figures.

[2-1. Voice synthesis method]
Fig. 6 is a sequence diagram showing a method for training an acoustic model and a method for synthesizing speech in an embodiment of the present invention. In the method for training an acoustic model shown in Fig. 6, a configuration is described in which information indicating the progress of a training job is made available to a third party from the time the execution of the training job is started at the instruction of a user until the trained acoustic model is completed. The steps before step S601 in Fig. 6 are the same as S401 to S403 in Fig. 4, and therefore will not be described. S601 in Fig. 6 is the same as S404 in Fig. 4. In the following description, a user who uses communication terminal 300A and corresponds to the third party may be referred to as a "second user".

Based on the execution instruction from the first user from the communication terminal 200A in S601, the server 100A starts the execution of the training job of the base acoustic model using the new sound source ID and the selected waveform set 149A (step S611). At the completion of the training job, the trained acoustic model 120A trained with this waveform set 149A is obtained as a result. When the training job is started in S611, the server 100A notifies the communication terminal 200A that the training job has been started, and inquires the communication terminal 200A about whether or not to disclose the status information indicating the status of the training job to a third party, that is, whether or not to allow a third party to view the status information (step S612). If the first user issues a disclosure instruction to disclose the status information indicating the status of the training job in response to the inquiry in S612, the communication terminal 200A transmits the disclosure instruction to the server 100A (step S602). If the first user does not issue a disclosure instruction, the communication terminal 200A does not transmit the disclosure instruction. This status information is sent to the communication terminal 200A regardless of whether or not a disclosure instruction has been issued, and is displayed on its display for viewing by the first user.

In S602, based on the disclosure instruction from the first user as described above, the server 100A discloses to the communication terminal 300A the status information indicating the status of the training job of the first user that was started in S611 (step S613). This allows a third party to view the status information displayed on the display of the communication terminal 300A.

Note that if the first user has agreed in advance to making the status information indicating the status of the training job public and has issued a disclosure instruction, steps S612 and S602 can be omitted. In other words, the status information indicating the status of the first user's training job may be made public to the second user based on the disclosure instruction issued in advance by the first user.

Steps S615 to S618 after S622 are similar to steps S414 to S417 in Figure 4, so their explanation will be omitted.

In FIG. 6, a configuration is illustrated in which a communication terminal 300A different from the communication terminal 200A that issued the instruction to execute the training job is the entity that executes the preview request, but this configuration is not limited to this. For example, the communication terminal 200A (first user) that issued the instruction to execute the training job may execute the preview request in order to check the progress of the training job itself. For example, by communication terminal 200A making a preview request, the training job can be ended at a timing when the first user is satisfied with the synthesized sound waveform for preview, even if the progress has not reached 100%.

[2-2. GUI 150A]
Here, an example of a GUI provided in S613 will be described. FIG. 7 is a diagram showing an example of a GUI related to disclosure of acoustic model information and a preview request in an embodiment of the present invention. is displayed on the display of communication terminal 200A, 300A.

As shown in FIG. 7, GUI 150A displays item 151A indicating the progress according to the status information, item 152A indicating detailed information, and preview button 157A for requesting preview. In this embodiment, item 151A indicating the progress indicates the progress of the training job of acoustic model 120A. However, item 151A may be an item other than the degree of completion, such as the elapsed time until a completion prediction of 100% is reached, and the degree of change in the variables of acoustic model 120A.

Item 151A is a progress bar that displays the progress of the training job as a percentage. In item 151A, the current state indicated by the progress may be the current training amount relative to the total training amount estimated at the start of the training job, or the current training amount relative to the total training amount estimated from the change in the variables of acoustic model 120A during the execution of the training job. In other words, the state of the training job changes over time, and server 100A provides and displays the progress indicating the change over time in the state of the training job as item 151A to the communication terminal. Since the state of the training job changes over time, server 100A repeatedly updates the state information indicating the state of the training job when the information changes or at regular intervals, and repeatedly provides it to communication terminals 200A and 300A.

In this embodiment, a configuration has been exemplified in which status information indicating the status of a training job is repeatedly provided to communication terminals 200A and 300A in real time, but this configuration is not limited to this. For example, the status information may be provided only once to each of communication terminals 200A and 300A. Alternatively, the status information may be displayed on communication terminal 300A (second device) based on a disclosure request made by a second user using communication terminal 300A, with the status information at the time of the disclosure request being displayed.

In FIG. 7, a progress bar is shown as an example of item 151A indicating the degree of progress, but the present invention is not limited to this configuration. For example, the degree of progress may be displayed as a numerical percentage.

Item 152A is information showing details of the training job. In FIG. 7, an acoustic model name 153A, a training sound waveform 154A, a completion forecast 155A, and a training executor 156A are displayed as examples of detailed information of item 152A. The acoustic model name 153A is a name set by the first user. For example, "voice X→Y" means that the pre-training acoustic model 120A (base acoustic model) that synthesizes the voice of X (one or more singers X, or one or more instruments X) is changed to a trained acoustic model 120A that synthesizes the voice of Y (a new singer Y or instrument Y) by the training job being executed. The training sound waveform 154A indicates a sound waveform used for training the acoustic model 120A in the training job being executed. The example in FIG. 7 means that the sound waveform B is used for the acoustic model 120A. The completion forecast 155A indicates the date and time when the progress of the training job being executed is expected to reach 100%. Training executor 156A indicates the name of the user who executed the ongoing training job. The user name may be an account name or a nickname. In FIG. 7, training executor 156A is "U1." U1 may be the same as or different from the singer or performer associated with Y.

Preview button 157A is a button for executing a preview request, which will be described later. For example, in FIG. 6, after the information is disclosed in S613, the second user presses preview button 157A, causing communication terminal 300A to request server 100A to preview the synthetic voice (step S621). When the preview request is executed in S621, server 100A executes preview speech synthesis using acoustic model 120A at the progress level at the time the preview request was executed, using the new sound source ID, and provides a synthetic voice waveform for preview (step S614). By providing the synthetic voice waveform for preview, communication terminal 300A can preview the synthetic voice generated by acoustic model 120A at the above time (step S622). Naturally, this preview can also be performed by communication terminal 200A.

Training jobs are executed in batches, with each batch being a group of processes. If acoustic model 120A is in the middle of a batch process at the time the preview request is executed, the synthetic sound waveform for preview generated by acoustic model 120A obtained in the immediately preceding batch process may be provided, or after that point, when the ongoing batch process is completed, the synthetic sound waveform for preview generated by the obtained acoustic model 120A may be provided. In other words, based on the preview request from communication terminals 200A and 300A, server 100A provides the first and second users with a synthetic sound waveform for preview generated by acoustic model 120A according to the timing of the preview request.

As described above, according to the acoustic model training system 10A of this embodiment, the second user of the communication terminal 300A can view the process in which the acoustic model 120A is trained and established by the training job. Alternatively, the first user of the communication terminal 200A can end the training job at a time when he or she is satisfied, even if the progress has not reached 100%, as described above.

[3. Third embodiment]
An acoustic model training system 10B according to the third embodiment will be described with reference to Figures 8 and 9. The overall configuration of the acoustic model training system 10B and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore their description will be omitted. In the following description, description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to Figures 1 to 5, and the alphabet "B" will be added after the reference numerals shown in these figures.

[3-1. Voice synthesis method]
Fig. 8 is a sequence diagram showing an acoustic model training method and a speech synthesis method in one embodiment of the present invention. In the acoustic model training method shown in Fig. 8, a first training job and a second training job are executed in parallel, and a configuration in which state information regarding each training job is selectively made public to a third party will be described. Steps before step S801 in Fig. 8 are the same as S401 to S403 in Fig. 4, and therefore description thereof will be omitted. S801 in Fig. 8 is the same as S404 in Fig. 4.

Based on the first execution instruction from the communication terminal 200B in S801, the server 100B executes a first training job of the first base acoustic model using the new sound source ID and the first waveform set selected by the first user (step S811). When the first training job is started in S811, the server 100B notifies the communication terminal 200B that the first training job has been started, and inquires of the communication terminal 200B as to whether or not the first status information related to the first training job should be made public to a third party (step S812). In this embodiment, the above "third party" corresponds to the second user. In response to the inquiry in S812, the communication terminal 200B transmits a disclosure instruction to the server 100B to disclose the first status information (step S802).

In S802, based on the first disclosure instruction by the first user as described above, the server 100B discloses the first status information regarding the first training job executed in S811 to the communication terminal 300B (second user) (step S813). If the first user does not issue the first disclosure instruction, the server 100B does not disclose the first status information to the second user.

Next, based on the second execution instruction from the communication terminal 200B in S803, the server 100B executes a second training job of the second base acoustic model using a new sound source ID and the second waveform set selected by the first user (step S814). The first training job and the second training job are executed in parallel by S811 and S814. The first base acoustic model and the second base acoustic model are independent of each other, and there is no correlation between the sound source IDs used by the two. For example, when n training jobs are processed in parallel, this is realized by starting n virtual machines. The second waveform set used in the second training job is different from the first waveform set used in the first training job, but the training program of the second training job is the same as the training program of the first training job. When the first training job is completed, the first trained acoustic model trained with the first waveform set is obtained as a result, and when the second training job is completed, the second trained acoustic model trained with the second waveform set is obtained as a result.

The method for executing the second training job is similar to the method for executing the first training job. In the second training job, a second waveform set is used, which is one or more sound waveforms selected by the first user from a plurality of pre-stored sound waveforms (and sound waveforms to be stored).

When the second training job is started in S814, the server 100B notifies the communication terminal 200B that the second training job has been started, and inquires of the communication terminal 200B whether or not to make the second status information related to the second training job public (step S815). In response to this inquiry, the communication terminal 200B transmits a second disclosure instruction to the server 100B to make the second status information related to the second training job public (step S804). The server 100B that has received the second disclosure instruction makes the second status information related to the second training job executed in S814 public to the communication terminal 300B (second user) (step S816). If the first user has not issued a second disclosure instruction, the server 100B does not make the second status information public to the second user.

Note that if the first user has agreed in advance to making the status information regarding the first training job or the second training job public and has issued a disclosure instruction, steps S812, S802, S815, and S804 can be omitted. In other words, the status information regarding the first training job or the second training job may be made public to the second user based on the disclosure instruction issued in advance by the first user.

Steps S831 to S821 after S816 are basically the same as steps S621 to S618 in FIG. 6, but are executed separately for each of the first training job and the second training job.

[3-2. GUI 160B]
Here, an example of a GUI provided to the first user in S815 will be described. Fig. 9 is a diagram showing an example of a public setting GUI when setting public information during training of an acoustic model in an embodiment of the present invention. The GUI 160B shown in Fig. 9 is displayed on the display of the communication terminal 200B of the first user.

As shown in FIG. 9, GUI 160B is a screen for setting what information to make public when making the status information of a training job public. In this embodiment, the public setting item 161B includes a first training job item 162B and a second training job item 167B. In the first training job 162B, items such as acoustic model name 163B, training sound waveform 164B, completion forecast 165B, and training executor 166B are displayed as examples of detailed settings. In the second training job 167B, items such as acoustic model name 168B, training sound waveform 169B, completion forecast 170B, and training executor 171B are displayed as examples of detailed settings. The above items are the same as the items shown in FIG. 7, so explanations will be omitted.

In GUI 160B of FIG. 9, items selected by the user are displayed as a filled square (■), and items not selected by the user are displayed as an open square (□). When an item of first training job 162B is selected by the first user, all detailed items related to the first training job are automatically selected. In this case, all items related to the first training job are made public. If an item of second training job 167B is not selected, the first user can individually select detailed items related to the second training job. In the case of FIG. 9, only the items acoustic model name 168B and training sound waveform 169B are selected. In this case, only the selected detailed items for the second training job are made public. The first communication terminal transmits a first disclosure instruction to the server 100B for the range of information selected by the first user as the disclosure target among the first status information of the first training job (S802 and S804), and transmits a second disclosure instruction for the range of information selected by the first user as the disclosure target among the second status information of the second training job (S804). In other words, the server 100B individually and selectively discloses at least one of the first status information and the second status information to the second user (provides it to the communication terminal 300B) based on the disclosure instruction from the first user. For items of the first training job and the second training job for which a disclosure instruction has not been received, the corresponding status information is not disclosed to the second user.

Note that in S812, a GUI similar to the above is provided, but in that GUI, only items related to the first training job 162B are displayed.

The publish button 172B is a button that instructs the disclosure of information about the acoustic model being trained. In S804 of FIG. 8, when the first user presses the publish button 172B, an instruction to publish items to be published selected by the user from among the status information of the first training job and the second training job is sent from the communication terminal 200B to the server 100B, and the status information of the items to be published is published to third parties in a format similar to that of FIG. 7 (step S816).

As described above, according to the acoustic model training system 10B of this embodiment, the first user can individually make multiple training jobs that the first user has started public to third parties. In addition, the first user can freely set which items to make public and which items not to make public for each detailed item of the training job.

[4. Fourth embodiment]
An acoustic model training system 10C according to the fourth embodiment will be described with reference to FIG. 10. The overall configuration of the acoustic model training system 10C and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore their description will be omitted. In the following description, description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to FIGS. 1 to 5, and the alphabet "C" will be added after the reference numerals shown in these figures.

[4-1. Voice synthesis method]
Fig. 10 is a flowchart showing a method for training an acoustic model in one embodiment of the present invention. In the method for training an acoustic model shown in Fig. 10, a training job instructed by a user is executed on the condition that the user has paid the charge. Fig. 10 explains the operations performed from the instruction of a training job in S404 in Fig. 4 to the execution of the training job in S413. Steps S1001 and S1004 in Fig. 10 are the same as S404 and S413 in Fig. 4, respectively.

As shown in FIG. 10, in S1001, communication terminal 200C transmits an instruction to execute a training job (first execution instruction) to server 100C. Next, server 100C, which has received the execution instruction, charges the first user who instructed the execution of the training job, and notifies communication terminal 200C of information related to the charge (step S1002). After the notification, server 100C determines whether communication terminal 200C has paid the charge to the operator of server 100C (step S1003). When communication terminal 200C has made the payment ("Yes" in S1003), server 100C executes the instructed training job on the base acoustic model using the selected waveform set, within the scope of the charge (step S1004). On the other hand, if the communication terminal 200C does not make the payment ("No" in S1003), the training job is not executed by the server 100C, and an error (non-execution of the training job) is notified to the communication terminal 200C (step S1005). The server 100C executes the billing process of S1002 each time the control unit of the server 100C executes a training job for a unit time (S1004), and when payment is received from the first user (S1003), the server 100C may execute the training job for the next unit time for the acoustic model being trained (S1004).

As described above, according to the acoustic model training system 10C of this embodiment, the first user can have the server 100C execute a training job that corresponds to the amount paid.

[5. Fifth embodiment]
An acoustic model training system 10D according to the fifth embodiment will be described with reference to Figures 11 to 14. The overall configuration of the acoustic model training system 10D and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore their description will be omitted. In the following description, description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to Figures 1 to 5, and the alphabet "D" will be added after the reference numerals shown in these figures.

[5-1. Voice synthesis method]
Fig. 11 is a sequence diagram showing a method for recording a sound waveform used for training an acoustic model in one embodiment of the present invention. In the recording method shown in Fig. 11, a configuration is described in which a training sound waveform is recorded in a recording space such as a karaoke box and uploaded to a server. The recording space is a real space. In the following description, a rental space is exemplified as the recording space.

The karaoke server 500D shown in FIG. 11 is, for example, a server or computer that manages the rental of karaoke boxes and karaoke booths. The karaoke server 500D manages a space ID that identifies one of multiple rental spaces, such as karaoke boxes and karaoke booths, provided in one store, and the availability that indicates whether each rental space is available. The rental space may be a completely closed space, such as a karaoke box, or a space that is partially open to the outside, such as a karaoke booth. Each rental space is equipped with karaoke equipment that has a recording function and a communication function with the karaoke server 500D. The karaoke server 500D can be connected to the network 400D and can communicate with the server 100D via the network 400D. In this embodiment, the server 100D handles the reservation of rental spaces for the karaoke server 500D. However, the details will be described later, but the present invention is not limited to this configuration.

First, the communication terminal 200D logs in to the acoustic model training service provided by the server 100D (step S1101). In S1101, the communication terminal 200D transmits to the server 100D account information (e.g., a user ID and a password) entered by a first user who uses the service. The server 100D performs user authentication based on the account information received from the communication terminal 200D, and approves the first user's login to the account with that user ID (step S1111). User authentication may be performed by an external authentication server instead of the server 100D.

The communication terminal 200D requests the reservation of a rental space of a desired space ID at a desired date and time including the use of the service, using the user ID logged in at S1111 (step S1102). When the server 100D receives the reservation request of S1102, it checks the usage status or availability of the rental space of the space ID at the date and time with the karaoke server 500D (step S1112). If the rental space is available, the karaoke server 500D makes a reservation (step S1121) and transmits reservation completion information to the server 100D indicating that the reservation of the rental space of the space ID at the date and time has been completed. If the first user has specified prepayment in the reservation request, the rental fee and the service fee are charged in step S1121. The service fee is the price for a basic training job using the waveform recorded there, which is executed after the rental space is used. The communication terminal 200D may make a reservation request for a rental space to the karaoke server, and in response to the reservation request, the karaoke server 500D that made the reservation may transmit reservation completion information including the user ID and space ID related to the reservation to the server 100D.

When the server 100D receives reservation completion information from the karaoke server 500D (step S1113), it links the space ID related to the reservation completion information with the user ID of the first user (step S1114). Then, it notifies the communication terminal 200D that the reservation has been completed (step S1115). The reservation completion notification may be sent from the karaoke server 500D to the communication terminal 200D.

When the communication terminal 200D receives the reservation completion notification, the communication terminal 200D displays to the first user that the reservation has been completed, as well as information identifying the reserved rental space and date and time. The information identifying the rental space is, for example, the room number of a karaoke booth identified by a space ID. When the first user moves to the reserved rental space on the reserved date and time and operates the karaoke equipment provided in the rental space to select a desired song, the accompaniment to the song is played in the rental space. The first user executes a recording start instruction and a recording end instruction using the karaoke equipment. In response to these instructions, the karaoke server 500D records the first user's singing voice or the sound of an instrument being played (step S1122).

When the usage time of the rental space ends (recording is completed), the karaoke server 500D (rental company) charges the first user for the rental space and training job usage fee if they have not been paid in advance, and the first user pays the usage fee at the terminal of the karaoke server 500D. Since the fee is set together with the rental fee, the fee for the training job may be discounted from the fee charged in S1002 accordingly. The first user selects a sound waveform to be uploaded to the server 100D from the sound waveforms (waveform data) that have been recorded, and further, if the fee for the training job has been paid, selects a waveform set to be used for the training job from the sound waveforms to be uploaded. The karaoke server 500D uploads the selected sound waveform and the space ID where the recording was performed to the memory area of the first user identified by the user ID of the first user of the server 100D (step S1123).

The server 100D stores the uploaded sound waveform and space ID in the memory area of the first user by linking them to each other (step S1116). The sound waveforms uploaded and stored in the server 100D may be one or multiple.

In S1114, the space ID and the user ID of the first user are linked, and in S1116, the uploaded sound waveform and the space ID are linked. Therefore, as shown in FIG. 12, the server 100D links and stores the user ID 180D of the first user, the space ID 181D, and the uploaded sound waveform 182D. FIG. 12 is an example of data managed by a server in one embodiment of the present invention. The user ID 180D is the user ID of the account logged in in S1111 of FIG. 11, and each data in FIG. 13 described later is stored in a storage area corresponding to the user ID. The space ID 181D is the space ID of the space where the recording was made in S1122 of FIG. 11. The sound waveform 182D is the sound waveform recorded in S1122 of FIG. 11 and sent to the server 100D in S1123.

The server 100D identifies the user ID of the first user who uploaded the sound waveform from the storage area where the sound waveform was uploaded in S1123 (step S1117). After that, based on an instruction from the first user, the server 100D executes a training job for the base acoustic model using the new sound source ID and the uploaded sound waveform (step S1118).

The data uploaded from the karaoke server 500D to the server 100D in S1123 will now be described with reference to FIG. 13. In the description of FIG. 11, a configuration has been exemplified in which only the sound waveform representing the first user's singing voice or performance sound is uploaded to the server 100D in S1123, but this configuration is not limiting. For example, in the case of singing voice, as shown in FIG. 13, pitch data 503D indicating the sounds constituting the guide melody of the song supplied to the rental space by the karaoke device and text data 502D indicating the lyrics of the song may be uploaded to the server 100D together with the sound waveform 501D. In the case of performance sound, the text data 502D is not uploaded.

The step in which the karaoke server 500D uploads the data recorded in S1122 to the server 100D in S1123 will be described with reference to FIG. 14. In the description of FIG. 11, a configuration in which the sound waveform recorded in S1122 is uploaded to the server 100D in S1123 without going through any particular steps has been exemplified, but this configuration is not limited to this. For example, as shown in FIG. 14, the first user may determine whether or not to upload the sound waveform after playing back the audio data related to the recorded sound waveform. In the example of FIG. 14, the karaoke machine or communication terminal 200D is used to inquire of the first user whether or not to play back the recorded sound waveform, whether or not to upload the sound waveform, whether or not to re-record, and whether or not to end the operation. These four inquiries may be displayed in sequence on one GUI, or may be displayed in parallel on the GUI as a play button, an upload button, a re-record button, and an end button.

After the recording of the audio data is completed in S1122 of FIG. 11, as shown in FIG. 14, the karaoke server 500D judges whether or not a playback instruction has been issued by the first user (step S1401). If a playback instruction has been issued in S1401 ("Yes" in S1401), the karaoke server 500D uses the karaoke equipment to play back the audio data recorded in S1122 of FIG. 11 in the rental space where the recording was made (step S1402). During the playback, the audio data alone may be played back, or the audio data may be played back together with a guide melody. After playback is performed in S1402, the process returns to step S1401. If no playback instruction has been issued in S1401 ("No" in S1401), the process proceeds to the next step without executing playback in S1402.

Next, a determination is made as to whether or not uploading is required for the voice data recorded in S1122 of FIG. 11 (step S1403). For example, the karaoke server 500D provides the first user with a GUI for selecting whether or not to upload the recorded voice data, and determines whether or not uploading is required based on the selection made by the first user.

If it is determined in S1403 that uploading is necessary ("Yes" in S1403), the upload in S1123 of FIG. 11 is executed, and the above operation ends. On the other hand, if there is no instruction to execute uploading in S1403 ("No" in S1403), a determination is made as to whether or not re-recording is necessary (step S1404). For example, the karaoke server 500D provides the first user with a GUI to select whether or not to re-record, and determines whether or not re-recording is necessary according to the selection made by the first user.

If it is determined in S1404 that re-recording is necessary ("Yes" in S1404), the karaoke server 500D re-records in the same manner as S1122 in FIG. 11 (step S1405). When the re-recording in S1405 is finished, the presence or absence of a playback instruction is determined again in S1401. If there is no instruction to start re-recording in S1404 ("No" in S1404), it is determined whether or not to end the operation (step S1406). If it is determined in S1406 that the operation can be ended ("Yes" in S1406), the above operation ends. On the other hand, if there is no instruction to end the operation in S1406 ("No" in S1406), the process returns to step S1401. If there is no instruction to play in S1401, an instruction to perform uploading in S1403, an instruction to start re-recording in S1404, or an instruction to end in S1406, the karaoke server 500D repeatedly executes these determination steps.

In this embodiment, the server 100D performs the reservation service for the rental space on behalf of the karaoke server 500D, but the present invention is not limited to this configuration. For example, the karaoke server 500D may perform the reservation service for the rental space. In this case, the server 100D and the karaoke server 500D share the first account information of the first user. Furthermore, the server 100D links the space ID and sound waveform received from the karaoke server 500D to the user ID (first account information) of the first user and stores them. Subsequent steps are the same as those from S1122 onwards in FIG. 11.

The recording start instruction and recording end instruction in S1122 in FIG. 11 may be executed by the start and end of the song, or by any operation of the first user. In other words, only audio data for a specified period of the song's playback period may be recorded based on the recording instruction of the first user. The recording start instruction and recording end instruction may be executed using a karaoke machine or may be executed using the communication terminal 200D. In other words, the recording in S1122 may be executed only for at least a part of the song's playback period. In other words, as shown in FIG. 13, the server 100D may receive from the karaoke server 500D pitch data 503D indicating the sound of the part of the song sung or played by the first user provided in the rental space and text data 502D indicating the lyrics of the song, together with sound waveforms 501D that are audio data recording singing during at least a part of the song's playback period. The server 100D then stores the sound waveforms 501D of the singing or playing sounds as training sound waveforms, linked to the sheet music data.

As described above, according to the acoustic model training system 10D of this embodiment, voice data can be recorded using a karaoke booth or the like and uploaded to the server 100D, thereby reducing the effort required for the first user to prepare an environment for recording voice data.

[6. Sixth embodiment]
An acoustic model training system 10E according to the sixth embodiment will be described with reference to FIG. 15. The overall configuration of the acoustic model training system 10E and the block diagram relating to the server are the same as those of the acoustic model training system 10 according to the first embodiment, and therefore their description will be omitted. In the following description, description of the same configuration as in the first embodiment will be omitted, and differences from the first embodiment will be mainly described. In the following description, when describing the same configuration as in the first embodiment, reference will be made to FIGS. 1 to 5, and the alphabet "E" will be added after the reference numerals shown in these figures.

[6-1. Voice synthesis method]
Fig. 15 is a flowchart showing a method for recommending a piece of music suitable for training a target acoustic model in one embodiment of the present invention. The recommendation method shown in Fig. 15 describes a configuration for recommending to a first user a piece of music suitable for a sound waveform based on all or a part of a sound waveform previously stored in the server 100E as a training sound waveform, or a waveform set selected by the user. The communication terminal 100E has received information from the first user in advance indicating the range of use of the acoustic model with respect to pitch or acoustic features that the first user expects.

First, the server 100E analyzes the previously stored training sound waveforms or the selected waveform set (step S1501). The training sound waveforms to be analyzed are not all of the stored training sound waveforms, but are part of a specific sound source (a specific singer or a specific instrument). For example, a folder for each singer or instrument may be provided in the memory area of the first user of the server 100E, and the training sound waveforms may be stored separately in folders for the corresponding singer or instrument, and the analysis may be performed individually for the sound waveforms stored in each folder. The waveform set is a set of sound waveforms for a specific singer or a specific instrument selected by the first user to train the acoustic model of a specific singer or a specific instrument. The analysis is performed, for example, based on the pitch or acoustic features of the sound waveform. Furthermore, if the song of the analyzed sound waveform is known, the singing skill or playing skill can be determined in terms of pitch, timbre, dynamics, etc. by comparing the sound waveform with the score data of the sung or played sound of the song. Alternatively, the analysis can determine the singing style, playing style, singing range, or playing range.

A singing style is a way of singing, and a performance style is a way of performing. Specifically, singing styles include neutral, vibrato, husky, fly, and growl. Performance styles include neutral, vibrato, pizzicato, spiccato, flageolet, and tremolo for bowed string instruments, and neutral, position, legato, slide, and slap/mute for plucked string instruments. For a clarinet, neutral, staccato, vibrato, and trill are examples. Note that, for example, the above vibrato refers to a singing style or performance style that uses vibrato a lot. The pitch, volume, timbre, and their dynamic behaviors in singing or performing vary depending on the style as a whole. In a training job, the server 100E may train the base acoustic model 120E while inputting a singing style or performance style obtained by analyzing the waveform set in addition to a new timbre ID and waveform set.

The singing range and performance range of the training sound waveforms are determined from the distribution of pitches in multiple sound waveforms of a specific singer's singing and a specific instrument's performance, and indicate the range of the sound waveform of that singer or instrument.

When the planned range of use of pitch data and acoustic features is not covered for the timbre of a specific sound source, the server 100E determines that the prepared training sound waveforms are not sufficient for training the acoustic model. By performing the analysis of S1501, the server 100E detects ranges for which there are no or few sound waveforms among the total range for which the timbre of a specific sound source is planned to be used. The server 100E then identifies one or more songs to recommend to the first user in order to fill in the ranges for which there is insufficient data (step S1502). The server 100E then provides information indicating the songs identified in S1502 to the communication terminal 200E (first user), and the communication terminal 200E displays the received information on its display.

As described above, according to the acoustic model training system 10E of this embodiment, if the sound waveforms prepared as training sound waveforms do not cover the intended range of use, the first user is notified of this, so that the first user can prepare training sound waveforms that cover the intended range of use.

The present invention is not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit and scope of the invention. The embodiments can be combined with each other as long as no technical contradiction occurs.

10: Acoustic model training system, 100: Server, 101: Control unit, 102: RAM, 103: ROM, 104: User interface, 105: Communication interface, 110: Storage, 111: Voice synthesis program, 112: Training job, 113: Music score data, 114: Sound waveform, 120: Acoustic model, 130: Synthesized sound waveform, 140: GUI, 141, 142, 143: Checkbox, 144: Execute button, 150A: GUI, 151A, 152A: Items indicating progress, 153A: Acoustic model name, 154A: Training sound waveform, 155A: Completion forecast, 156A: Training executor, 157A: Preview button, 160B: GUI, 161B: Public setting item, 162B: First training job, 163B, 168B: Acoustic model name, 164B, 169B: Training sound waveform, 165B, 170B: Completion forecast, 166B, 171B: Training executor, 167B: Second training job, 172B: Publish button, 180D: Account information, 182D: Sound waveform, 200, 300: Communication terminal, 400: Network, 411: Step, 500D: Karaoke server, 501D: Sound waveform, 502D: Text data, 503D: Pitch data

Claims (19)

a first device connectable to a network and used by a first user;
a server connectable to the network;
The first device, under control of the first user,
uploading a plurality of sound waveforms to the server;
selecting one or more sound waveforms as a first waveform set from the plurality of sound waveforms already uploaded or to be uploaded;
Transmitting a first execution instruction to the server for a first training job for an acoustic model that generates acoustic features;
The server, based on the first execution instruction from the first device,
commencing execution of the first training job using the selected first waveform set;
An acoustic model training system that provides a trained acoustic model trained by the first training job to the first device. A method for training an acoustic model implemented by one or more computers, which provides a first user with an interface that allows the user to select one or more sound waveforms to be used in a first training job for an acoustic model that generates acoustic features from a plurality of pre-stored sound waveforms. receiving one or more waveforms selected by the first user using the interface as a first waveform set;
commencing execution of the first training job using the first waveform set based on a first execution instruction by the first user via the interface;
The training method of claim 2 , further comprising: providing the acoustic model trained by the first training job to the first user as a first acoustic model. The training method according to claim 3, further comprising providing first status information indicating a status of the first training job to a second user different from the first user based on a first disclosure instruction by the first user. Displaying the first status information on a first device used by the first user;
The training method of claim 4 , further comprising: displaying the first status information on a second device used by the second user based on the first disclosure instruction. The state of the first training job changes over time;
The training method of claim 4 , wherein the first status information displayed on a second device used by the second user is updated repeatedly. The training method according to claim 4, wherein the progress of the state of the first training job is displayed as the first status information. The training method according to claim 4, wherein the first status information regarding the timing of the disclosure request is displayed on a second device used by the second user based on a disclosure request by the second user. receiving a second set of waveforms, the second set being a new selection of one or more waveforms by the first user using the interface;
and initiating execution of a second training job using the second waveform set based on a second execution instruction by the first user.
The training method of claim 3 , wherein the first training job and the second training job are performed in parallel. The training method according to claim 9, further comprising providing at least one of first status information regarding the first training job and second status information regarding the second training job to a second device of a second user different from the first user, based on a disclosure instruction by the first user. billing the first user in response to the first execution instruction of the first user;
The method of claim 2 , further comprising starting execution of the first training job if payment of the charge is confirmed. receiving a space ID that identifies a real space;
The training method according to claim 2 , further comprising linking the space ID with account information of the first user for a service that provides the training method. The training method according to claim 12, further comprising: charging the first user who has the account information linked to the space ID. receiving sheet music data representing the sounds constituting the piece of music reproduced in the real space together with audio data recording the singing or playing sounds during at least a portion of the reproduction period of the piece of music;
13. The training method according to claim 12, wherein the voice data is stored as a pre-saved sound waveform in a manner linked to the musical score data. The training method according to claim 14, wherein the audio data is recorded for a specified period of the playback period based on a recording instruction from the first user. reproducing the audio data in the real space based on a reproduction instruction from the first user;
The training method according to claim 14, further comprising the step of inquiring the first user as to whether or not to save the audio data played back in response to the playback instruction as one of the plurality of pre-stored sound waveforms provided to the first user. Analyzes pre-stored sound waveforms,
Identifying a piece of music to be recommended to the first user based on an analysis result obtained by the analysis;
The method of claim 2 , further comprising providing information to the first user indicative of the identified piece of music. The training method according to claim 17, wherein the analysis results indicate at least one of singing style, playing style, singing range, and playing range. The training method according to claim 17, wherein the analysis results indicate playing skills.

Images