JP7424359B2 - Information processing device, singing voice output method, and program - Google Patents

Description

The present invention relates to a technique for responding to user input using voice, including singing.

2. Description of the Related Art Techniques for outputting music according to user instructions are known. For example, Patent Document 1 discloses a technique for changing the atmosphere of a song according to the user's situation and preferences. Patent Document 2 discloses a technique for selecting a unique piece of music that you will never get tired of in a device that outputs musical tones depending on the state of a moving body.

Japanese Patent Application Publication No. 2006-85045 Patent No. 4496993

In both Patent Documents 1 and 2, singing voices are not output in response to interaction with a user.
In contrast, the present invention provides a technique for outputting singing voices in response to interaction with a user.

The present invention includes a step of specifying a first partial content from among a plurality of partial contents obtained by decomposing a character string included in the content, and a step of synthesizing the content using the character string included in the first partial content. a step of outputting the singing voice, a step of receiving a user's reaction to the singing voice, and a step of synthesizing the voice using a character string included in a second partial content following the first partial content according to the reaction. Provided is a method for outputting a singing voice, the method comprising:

This singing voice output method may include a step of determining, depending on the reaction, elements to be used for singing synthesis using character strings included in the second partial content.

The elements may include parameters of the singing synthesis, a melody, or a tempo, or an arrangement of accompaniment in the singing voice.

The synthesis of the singing voice is performed using the segments recorded in at least one database selected from a plurality of databases, and the method of outputting the singing voice is based on the second The method may include a step of selecting a database to be used in song synthesis using character strings included in the partial content.

The synthesis of the singing voice is performed using the fragments recorded in a plurality of databases selected from among the plurality of databases, and in the step of selecting the database, the plurality of databases are selected and the synthesis of the singing voice is performed. The output method may include a step of determining a utilization ratio of the plurality of databases according to the reaction.

This method of outputting singing audio includes the step of replacing a part of the character string included in the first partial content with another character string, and in the step of outputting the singing audio, a part of the character string is replaced with another character string. A singing voice synthesized using a character string included in the first partial content replaced with a character string may be output.

The other character string and the character string to be replaced may have the same number of syllables or the same number of moras.

This method of outputting a singing voice includes the step of replacing a part of the second partial content with another character string in accordance with the reaction, and in the step of outputting the singing voice, a part of the second partial content is replaced with another character string. A singing voice synthesized using the character string included in the second partial content replaced with the character string may be output.

This method of outputting a singing voice combines the singing voice of the first partial content with the singing voice of the second partial content, which is synthesized so as to have a time length corresponding to the item indicated by the character string included in the first partial content. It may also include a step of outputting the partial content between the singing voice and the singing voice.

This method of outputting a singing voice is such that a singing voice synthesized using a second character string corresponding to a matter indicated by a first character string included in the first partial content is outputted to the singing voice of the first partial content. After outputting, the first character string may include a step of outputting at a timing corresponding to a time length corresponding to the item indicated by the first character string.

The present invention also provides a specifying unit that identifies a first partial content from among a plurality of partial contents obtained by disassembling a character string included in the content; an output unit that outputs a singing voice synthesized using the singing voice; and a receiving unit that receives a user's reaction to the singing voice; To provide an information processing system that outputs a singing voice synthesized using character strings included in partial content No. 2.

According to the present invention, singing voice can be output in response to interaction with a user.

FIG. 1 is a diagram showing an overview of a voice response system 1 according to an embodiment. 1 is a diagram illustrating an overview of functions of a voice response system 1. FIG. 1 is a diagram illustrating a hardware configuration of an input/output device 10. FIG. FIG. 3 is a diagram illustrating the hardware configuration of a response engine 20 and a singing synthesis engine 30. 5 is a diagram illustrating a functional configuration related to a learning function 51. FIG. 5 is a flowchart showing an overview of operations related to the learning function 51. 5 is a sequence chart illustrating an operation related to the learning function 51. A diagram illustrating a classification table 5161. A diagram illustrating a functional configuration related to a singing synthesis function 52. 5 is a flowchart showing an overview of operations related to the singing synthesis function 52. 5 is a sequence chart illustrating operations related to the singing synthesis function 52. 5 is a diagram illustrating a functional configuration related to a response function 53. FIG. 5 is a flowchart illustrating an operation related to the response function 53. FIG. 2 is a diagram showing a first example of operation of the voice response system 1. FIG. FIG. 2 is a diagram illustrating a second example of operation of the voice response system 1; FIG. 3 is a diagram showing a third example of operation of the voice response system 1; FIG. 4 is a diagram showing a fourth example of operation of the voice response system 1; FIG. 5 is a diagram showing a fifth example of operation of the voice response system 1; FIG. 6 is a diagram showing a sixth example of operation of the voice response system 1; FIG. 7 is a diagram showing a seventh example of operation of the voice response system 1; FIG. 7 is a diagram showing an example 8 of operation of the voice response system 1; FIG. 9 is a diagram showing a ninth example of operation of the voice response system 1; FIG. 3 is a diagram showing an example 10 of operation of the voice response system 1; FIG. 3 is a diagram showing an example 11 of operation of the voice response system 1;

1. System Overview FIG. 1 is a diagram showing an overview of a voice response system 1 according to an embodiment. The voice response system 1 is a system that automatically outputs a voice response when a user inputs (or gives instructions) by voice, and is a so-called AI (Artificial Intelligence) voice assistant. Hereinafter, the voice input from the user to the voice response system 1 will be referred to as "input voice", and the voice output from the voice response system 1 in response to the input voice will be referred to as "response voice". In this particular example, the audio response includes singing. That is, the voice response system 1 is an example of a singing synthesis system. For example, when a user speaks to the voice response system 1 by saying, "Sing something," the voice response system 1 automatically synthesizes a song and outputs the synthesized song.

The voice response system 1 includes an input/output device 10, a response engine 20, and a singing synthesis engine 30. The input/output device 10 is a device that provides a man-machine interface, and is a device that receives input voice from a user and outputs a response voice to the input voice. The response engine 20 analyzes the input voice received by the input/output device 10 and generates a response voice. This response voice includes singing voice at least in part. The singing synthesis engine 30 synthesizes singing voices used as response voices.

FIG. 2 is a diagram illustrating an overview of the functions of the voice response system 1. The voice response system 1 has a learning function 51, a singing synthesis function 52, and a response function 53. The response function 53 is a function that analyzes the user's input voice and provides a response voice based on the analysis result, and is provided by the input/output device 10 and the response engine 20. The learning function 51 is a function of learning the user's preferences from the user's input voice, and is provided by the singing synthesis engine 30. The singing synthesis function 52 is a function of synthesizing singing voices used as response voices, and is provided by the singing synthesis engine 30. The relationship between the learning function 51, the singing synthesis function 52, and the response function 53 is as follows. The learning function 51 uses the analysis results obtained by the response function 53 to learn the user's preferences. The singing synthesis function 52 synthesizes singing voices based on the learning performed by the learning function 51. The response function 53 responds using the singing voice synthesized by the singing synthesis function 52. Details of each function will be described later.

FIG. 3 is a diagram illustrating the hardware configuration of the input/output device 10. As shown in FIG. The input/output device 10 includes a microphone 101, an input signal processing section 102, an output signal processing section 103, a speaker 104, a CPU (Central Processing Unit) 105, a sensor 106, a motor 107, and a network IF 108. The microphone 101 is a device that converts a user's voice into an electrical signal (input sound signal). The input signal processing unit 102 is a device that performs processing such as analog/digital conversion on an input sound signal and outputs data representing input sound (hereinafter referred to as "input sound data"). The output signal processing unit 103 is a device that performs processing such as digital/analog conversion on data indicating a response voice (hereinafter referred to as "response voice data") and outputs an output sound signal. The speaker 104 is a device that converts an output sound signal into sound (outputs sound based on the output sound signal). The CPU 105 is a device that controls other elements of the input/output device 10, and reads programs from a memory (not shown) and executes them. The sensor 106 is a sensor that detects the user's position (the user's direction as seen from the input/output device 10), and is, for example, an infrared sensor or an ultrasonic sensor. The motor 107 changes the direction of at least one of the microphone 101 and the speaker 104 so as to face the direction in which the user is present. In one example, the microphone 101 may be a microphone array, and the CPU 105 may detect the direction in which the user is located based on sounds picked up by the microphone array. The network IF 108 is an interface for communicating via a network (for example, the Internet), and includes, for example, an antenna and a chipset for communicating in accordance with a predetermined wireless communication standard (for example, so-called WiFi (registered trademark)). include.

FIG. 4 is a diagram illustrating the hardware configuration of the response engine 20 and the singing synthesis engine 30. The response engine 20 is a computer device having a CPU 201, a memory 202, a storage 203, and a communication IF 204. The CPU 201 performs various calculations according to programs and controls other elements of the computer device. The memory 202 is a main storage device that functions as a work area when the CPU 201 executes a program, and includes, for example, RAM (Random Access Memory). The storage 203 is a nonvolatile auxiliary storage device that stores various programs and data, and includes, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Communication IF 204 includes a connector and a chipset for communicating in accordance with a predetermined communication standard (eg, Ethernet). In this example, the storage 203 stores a program (hereinafter referred to as a "response program") for causing a computer device to function as the response engine 20 in the voice response system 1. The computer device functions as the response engine 20 by the CPU 201 executing the response program. The response engine 20 is, for example, a so-called AI.

The singing synthesis engine 30 is a computer device having a CPU 301, a memory 302, a storage 303, and a communication IF 304. The details of each element are the same as those of the response engine 20. In this example, the storage 303 stores a program (hereinafter referred to as "singing synthesis program") for causing the computer device to function as the singing synthesis engine 30 in the voice response system 1. When the CPU 301 executes the singing synthesis program, the computer device functions as the singing synthesis engine 30.

In this example, the response engine 20 and the singing synthesis engine 30 are provided as a so-called cloud service on the Internet. Note that the response engine 20 and the singing synthesis engine 30 may be services that are not based on cloud computing. The details and operation of each of the learning function 51, singing synthesis function 52, and response function 53 will be described below.

2. Learning function 2-1. Configuration FIG. 5 is a diagram illustrating a functional configuration related to the learning function 51. As shown in FIG. As functional elements related to the learning function 51, the voice response system 1 includes a voice analysis section 511, an emotion estimation section 512, a music analysis section 513, a lyrics extraction section 514, a preference analysis section 515, a storage section 516, and a processing section 510. . The input/output device 10 also functions as a reception unit that receives user's input voice and an output unit that outputs response voice.

The speech analysis unit 511 analyzes input speech. The analysis here refers to the process of acquiring information used to generate a response voice from the input voice. Specifically, it refers to the process of converting the input voice into text (that is, converting it into a character string), and the process of converting the input voice into a text string. A process of determining a user's request from a user's request, a process of identifying a content providing unit 60 that provides content in response to the user's request, a process of instructing the identified content providing unit 60, and acquiring data from the content providing unit 60. This includes processing to generate a response using the acquired data. In this example, the content providing unit 60 is an external system of the voice response system 1. The content providing unit 60 is a computer resource that provides at least a service (for example, a music streaming service or internet radio) that outputs data for reproducing content such as music as sound (hereinafter referred to as "song data"). , for example, an external server of the voice response system 1.

The music analysis unit 513 analyzes music data output from the content providing unit 60. Analysis of music data refers to processing for extracting features of music. The characteristics of a song include, for example, at least one of melody, rhythm, chord progression, tempo, and arrangement. A known technique is used to extract the features.

The lyrics extracting unit 514 extracts lyrics from the music data output from the content providing unit 60. In one example, music data includes metadata in addition to sound data. The sound data is data indicating a signal waveform of a music piece, and includes, for example, uncompressed data such as PCM (Pulse Code Modulation) data, or compressed data such as MP3 data. Metadata is data that includes information related to the song, and includes information such as the song title, performer name, composer name, lyricist name, album title, song attributes such as genre, and lyrics. . Lyrics extraction unit 514 extracts lyrics from metadata included in music data. If the music data does not include metadata, the lyrics extraction unit 514 performs voice recognition processing on the sound data and extracts lyrics from the text obtained by voice recognition.

The emotion estimation unit 512 estimates the user's emotion. In this example, the emotion estimation unit 512 estimates the user's emotion from the input voice. Known techniques are used to estimate emotions. In one example, the emotion estimation unit 512 may estimate the user's emotion based on the relationship between the (average) pitch of the voice output by the voice response system 1 and the pitch of the user's response thereto. Alternatively, the emotion estimation unit 512 may estimate the user's emotion based on the input voice converted into text by the voice analysis unit 511 or the user's request analyzed.

The preference analysis unit 515 indicates the user's preferences using at least one of the playback history, analysis results, and lyrics of the song that the user has instructed to play, and the user's emotion when the user has instructed to play the song. Generate information (hereinafter referred to as "preference information"). The preference analysis unit 515 updates the classification table 5161 stored in the storage unit 516 using the generated preference information. The classification table 5161 is a table (or database) that records user preferences, and includes, for example, song characteristics (for example, timbre, melody, rhythm, chord progression, and tempo) and song attributes for each user and emotion. (performer's name, composer's name, lyricist's name, and genre) and lyrics are recorded. The storage unit 516 is an example of a reading unit that reads parameters corresponding to the user who inputs the trigger from a table in which parameters used for singing synthesis are recorded in association with users. Note that the parameters used for singing synthesis refer to data referenced during singing synthesis, and in the example of the classification table 5161, the parameters include timbre, melody, rhythm, chord progression, tempo, performer name, composer name, This concept includes the lyricist name, genre, and lyrics.

2-2. Operation FIG. 6 is a flowchart showing an overview of the operation of the voice response system 1 related to the learning function 51. In step S11, the voice response system 1 analyzes the input voice. In step S12, the voice response system 1 performs processing instructed by the input voice. In step S13, the voice response system 1 determines whether the input voice includes items to be learned. If it is determined that the input voice includes items to be learned (S13: YES), the voice response system 1 moves the process to step S14. If it is determined that the input voice does not include the subject of learning (S13: NO), the voice response system 1 moves the process to step S18. In step S14, the voice response system 1 estimates the user's emotion. In step S15, the voice response system 1 analyzes the music piece that has been instructed to be played. In step S16, the voice response system 1 obtains the lyrics of the song for which playback has been instructed. In step S17, the voice response system 1 updates the classification table using the information obtained in steps S14 to S16.

Although the processing after step S18 is not directly related to the learning function 51, that is, updating the classification table, it will be explained because it includes processing using the classification table. In step S18, the voice response system 1 generates a response voice to the input voice. At this time, the classification table is referred to as necessary. In step S19, the voice response system 1 outputs a response voice. Hereinafter, the operation of the voice response system 1 related to the learning function 51 will be explained in more detail.

FIG. 7 is a sequence chart illustrating the operation of the voice response system 1 related to the learning function 51. A user performs user registration with the voice response system 1, for example, when joining the voice response system 1 or when starting the voice response system 1 for the first time. User registration includes, for example, setting a user name (or login ID) and password. At the start of the sequence in FIG. 7, the input/output device 10 is activated and the user login process is completed. That is, in the voice response system 1, the user using the input/output device 10 is specified. Further, the input/output device 10 is in a state of waiting for voice input (utterance) from the user. Note that the method by which the voice response system 1 identifies a user is not limited to the login process. For example, the voice response system 1 may identify the user based on the input voice.

In step S101, the input/output device 10 receives input audio. The input/output device 10 converts input audio into data and generates audio data. The audio data includes audio data indicating the signal waveform of input audio and a header. The header includes information indicating attributes of the input audio. The attributes of the input voice include, for example, an identifier for identifying the input/output device 10, a user identifier (for example, a user name or login ID) of the user who uttered the voice, and a timestamp indicating the time when the voice was uttered. include. In step S102, the input/output device 10 outputs audio data indicating input audio to the audio analysis unit 511.

In step S103, the voice analysis unit 511 analyzes the input voice using voice data. In this analysis, the speech analysis unit 511 determines whether the input speech includes items to be learned. In this example, the items to be learned refer to items that specify a song, and specifically, are instructions for playing a song.

In step S104, the processing unit 510 performs processing instructed by the input voice. The processing performed by the processing unit 510 is, for example, streaming playback of music. In this case, the content providing unit 60 has a music database in which a plurality of pieces of music data are recorded. The processing unit 510 reads music data of the instructed music from the music database. The processing unit 510 transmits the read music data to the input/output device 10 that is the source of the input audio. In another example, the processing performed by the processing unit 510 is playback of internet radio. In this case, the content providing unit 60 performs streaming broadcast of radio audio. The processing unit 510 transmits the streaming data received from the content providing unit 60 to the input/output device 10 that is the source of the input audio.

If it is determined in step S103 that the input speech includes items to be learned, the processing unit 510 further performs processing to update the classification table (step S105). In this example, the processing for updating the classification table includes a request to the emotion estimation unit 512 for emotion estimation (step S1051), a request to the music analysis unit 513 to analyze the song (step S1052), and a request to the lyrics extraction unit 514 to perform song analysis. It includes an extraction request (step S1053).

When emotion estimation is requested, the emotion estimation unit 512 estimates the user's emotion (step S106), and outputs information indicating the estimated emotion (hereinafter referred to as “emotion information”) to the processing unit 510 that is the request source. (Step S107). In this example, the emotion estimation unit 512 estimates the user's emotion using the input voice. The emotion estimating unit 512 estimates emotions based on, for example, input speech converted into text. In one example, a keyword indicating an emotion is defined in advance, and if the input voice converted into text includes this keyword, the emotion estimation unit 512 determines that the user has that emotion (for example, "Damn it"). If the keyword is included, it is determined that the user's emotion is "anger"). In another example, the emotion estimating unit 512 estimates the emotion based on the pitch, volume, and speed of the input voice, or temporal changes thereof. In one example, if the average pitch of the input voice is lower than the threshold, the emotion estimation unit 512 determines that the user's emotion is "sad." In another example, the emotion estimation unit 512 may estimate the user's emotion based on the relationship between the (average) pitch of the voice output by the voice response system 1 and the pitch of the user's response thereto. . Specifically, if the pitch of the voice output by the voice response system 1 is high but the pitch of the voice the user responds to is low, the emotion estimation unit 512 determines that the user's emotion is "sad". to decide. In yet another example, the emotion estimation unit 512 may estimate the user's emotion based on the relationship between the pitch of the final word in the voice and the pitch of the user's response to the pitch. Alternatively, the emotion estimation unit 512 may estimate the user's emotion by considering these multiple factors in a composite manner.

In another example, the emotion estimator 512 may estimate the user's emotion using input other than voice. As the input other than audio, for example, an image of the user's face captured by a camera, the user's body temperature detected by a temperature sensor, or a combination thereof is used. Specifically, the emotion estimation unit 512 determines whether the user's emotion is "happy," "angry," or "sad" from the user's facial expression. Furthermore, the emotion estimation unit 512 may determine the user's emotion based on changes in facial expressions in a video of the user's face. Alternatively, the emotion estimation unit 512 may determine that the user is "angry" if the user's body temperature is high, and that the user is "sad" if the user's body temperature is low.

When music analysis is requested, the music analysis unit 513 analyzes the music played according to the user's instructions (step S108), and sends information indicating the analysis results (hereinafter referred to as "music information") to the requesting process. The information is output to the unit 510 (step S109).

When lyrics extraction is requested, the lyrics extraction unit 514 acquires the lyrics of the song to be played according to the user's instructions (step S110), and sends information indicating the acquired lyrics (hereinafter referred to as "lyrics information") to the requester. is output to the processing unit 510 (step S111).

In step S<b>112 , the processing unit 510 outputs the set of emotion information, music information, and lyrics information obtained from the emotion estimation unit 512 , music analysis unit 513 , and lyrics extraction unit 514 to the preference analysis unit 515 .

In step S113, the preference analysis unit 515 analyzes multiple sets of information to obtain information indicating the user's preferences. For this analysis, the preference analysis unit 515 records a plurality of sets of these pieces of information over a certain period in the past (for example, a period from the start of operation of the system to the present time). In one example, the preference analysis unit 515 statistically processes the music information and calculates a statistical representative value (for example, an average value, a mode value, or a median value). Through this statistical processing, for example, the average value of tempo and the mode of tone, melody, rhythm, chord progression, composer's name, lyricist's name, and performer's name are obtained. In addition, the preference analysis unit 515 uses technology such as morphological analysis to decompose the lyrics indicated by the lyric information into word level, identifies the part of speech of each word, and creates a histogram for words with a specific part of speech (for example, a noun). words whose appearance frequency falls within a predetermined range (for example, the top 5%). Furthermore, the preference analysis unit 515 extracts a group of words that include the identified word and correspond to a predetermined syntactic break (for example, a minute, clause, or phrase) from the lyrics information. For example, if the word "suki" appears frequently, a group of words including this word, such as "I like you so much" and "I like you very much," are extracted from the lyrics information. These average values, mode values, and word groups are examples of information (parameters) indicating user preferences. Alternatively, the preference analysis unit 515 may analyze multiple sets of information according to a predetermined algorithm different from simple statistical processing to obtain information indicating the user's preferences. Alternatively, the preference analysis unit 515 may receive feedback from the user and adjust the weights of these parameters according to the feedback. In step S114, the preference analysis unit 515 updates the classification table 5161 using the information obtained in step S113.

FIG. 8 is a diagram illustrating the classification table 5161. This figure shows a classification table 5161 for a user whose user name is "Taro Yamada." In the classification table 5161, the characteristics, attributes, and lyrics of songs are recorded in association with the user's emotions. Referring to the classification table 5161, for example, when the user "Taro Yamada" is feeling "happy", the words "koi", "ai", and "love" are included in the lyrics, and the tempo is about 60. This indicates that he prefers songs that have a chord progression of "I→V→VIm→IIIm→IV→I→IV→V" and are dominated by piano tones. According to this embodiment, information indicating the user's preferences can be automatically obtained. The preference information recorded in the classification table 5161 is accumulated as learning progresses, that is, as the cumulative usage time of the voice response system 1 increases, and becomes more reflective of the user's preferences. According to this example, information that reflects the user's preferences can be automatically obtained.

Note that the preference analysis unit 515 may set the initial value of the classification table 5161 at a predetermined timing, such as at the time of user registration or first login. In this case, the voice response system 1 allows the user to select a character (for example, a so-called avatar) representing the user on the system, and creates a classification table 5161 having an initial value according to the selected character into a classification corresponding to the user. It may also be set as a table.

The data recorded in the classification table 5161 described in this embodiment is merely an example. For example, the classification table 5161 does not need to record the user's emotions, but only needs to record at least the lyrics. Alternatively, the lyrics may not be recorded in the classification table 5161, and at least the user's emotions and the results of song analysis may be recorded.

3. Singing synthesis function 3-1. Configuration FIG. 9 is a diagram illustrating a functional configuration related to the singing synthesis function 52. As functional elements related to the singing synthesis function 52, the voice response system 1 includes a voice analysis section 511, an emotion estimation section 512, a storage section 516, a detection section 521, a singing generation section 522, an accompaniment generation section 523, and a synthesis section 524. . The song generation section 522 includes a melody generation section 5221 and a lyrics generation section 5222. In the following, description of elements common to the learning function 51 will be omitted.

Regarding the singing synthesis function 52, the storage unit 516 stores a segment database 5162. The segment database is a database that records speech segment data used in singing synthesis. The speech segment data is data obtained by converting one or more phonemes. A phoneme corresponds to the smallest unit of linguistic meaning distinction (e.g. vowels and consonants), and is defined by the phonology of a language, which is established by considering the actual articulation and phonological system of the language as a whole. is the smallest unit of A speech segment is a segment corresponding to a desired phoneme or phoneme chain cut out of input speech uttered by a specific speaker. The speech segment data in this embodiment is data indicating the frequency spectrum of a speech segment. In the following description, the term "phonetic segment" includes a single phoneme (eg, monophone) and a chain of phonemes (eg, diphone or triphone).

The storage unit 516 may store a plurality of elemental piece databases 5162. The plurality of segment databases 5162 may include, for example, records of phonemes pronounced by different singers (or speakers). Alternatively, the plurality of segment databases 5162 may include records of phonemes pronounced in different singing styles or tones by a single singer (or speaker).

The singing generation unit 522 generates singing voices, that is, performs singing synthesis. Singing audio refers to audio that utters given lyrics according to a given melody. The melody generation unit 5221 generates a melody used for singing synthesis. The lyrics generation unit 5222 generates lyrics used for singing synthesis. The melody generation unit 5221 and the lyrics generation unit 5222 may generate the melody and lyrics using the information recorded in the classification table 5161. The singing generation unit 522 generates singing voice using the melody generated by the melody generation unit 5221 and the lyrics generated by the lyrics generation unit 5222. The accompaniment generation unit 523 generates accompaniment for the singing voice. The synthesis unit 519 synthesizes a singing voice using the singing voice generated by the singing generation unit 522, the accompaniment generated by the accompaniment generation unit 523, and the speech segments recorded in the segment database 5162.

3-2. Operation FIG. 10 is a flowchart showing an overview of the operation (singing synthesis method) of the voice response system 1 related to the singing synthesis function 52. In step S21, the voice response system 1 determines whether an event that triggers singing synthesis has occurred. That is, the voice response system 1 detects an event that triggers singing synthesis. The event that triggers singing synthesis is, for example, an event in which the user inputs a voice, an event registered in a calendar (for example, an alarm or the user's birthday), or an event in which the user inputs a voice using a method other than voice (for example, the input/output device 10). The event includes at least one of an event in which an instruction for singing synthesis is input by an operation on a smartphone (not shown) wirelessly connected to the computer, and an event that occurs randomly. If it is determined that an event that triggers singing synthesis has occurred (S21: YES), the voice response system 1 moves the process to step S22. If it is determined that an event that triggers singing synthesis has not occurred (S21: NO), the voice response system 1 waits until an event that triggers singing synthesis occurs.

In step S22, the voice response system 1 reads singing synthesis parameters. In step S23, the voice response system 1 generates lyrics. In step S24, the voice response system 1 generates a melody. In step S25, the voice response system 1 modifies one of the generated lyrics and melody to match the other. Step S2
In step 6, the voice response system 1 selects a segment database to be used. In step S27, the voice response system 1 performs singing synthesis using the melody, lyrics, and segment database obtained in steps S23, S26, and S27. In step S28, the voice response system 1 generates accompaniment. In step S29, the voice response system 1 synthesizes the singing voice and the accompaniment. The processing in steps S23 to S29 is a part of the processing in step S18 in the flow of FIG. Hereinafter, the operation of the voice response system 1 related to the singing synthesis function 52 will be explained in more detail.

FIG. 11 is a sequence chart illustrating the operation of the voice response system 1 related to the singing synthesis function 52. When detecting an event that triggers song synthesis, the detection unit 521 requests song synthesis to the song generation unit 522 (step S201). The song synthesis request includes the user's identifier. When song synthesis is requested, the song generation unit 522 inquires of the user's preferences from the storage unit 516 (step S202). This query includes a user identifier. Upon receiving the inquiry, the storage unit 516 reads preference information corresponding to the user identifier included in the inquiry from the classification table 5161, and outputs the read preference information to the song generation unit 522 (step S203). Further, the song generation unit 522 inquires the emotion estimation unit 512 about the user's emotion (step S204). This query includes a user identifier. Upon receiving the inquiry, the emotion estimation unit 512 outputs the user's emotion information to the song generation unit 522 (step S205).

In step S206, the song generation unit 522 selects a lyrics source. The source of the lyrics is determined according to the input audio. The source of lyrics is roughly either the processing unit 510 or the classification table 5161. The song synthesis request output from the processing unit 510 to the song generation unit 522 may include lyrics (or lyric material) or may not include lyrics. A lyric material is a character string that cannot form lyrics by itself, but can form lyrics by combining with other lyric materials. The case where the request for singing synthesis includes lyrics refers to, for example, a case where a melody is added to the response itself by the AI (such as "Tomorrow's weather will be sunny") and a response voice is output. Since the singing synthesis request is generated by the processing unit 510, it can also be said that the source of the lyrics is the processing unit 510. Furthermore, since the processing unit 510 may acquire content from the content providing unit 60, it can also be said that the source of the lyrics is the content providing unit 60. The content providing unit 60 is, for example, a server that provides news or a server that provides weather information. Alternatively, the content providing unit 60 is a server that has a database that records lyrics of existing songs. Although only one content providing unit 60 is shown in the figure, a plurality of content providing units 60 may exist. If the song synthesis request includes lyrics, the song generation unit 522 selects the song synthesis request as the lyrics source. If the request for singing synthesis does not include lyrics (for example, if the input voice instruction does not specify the content of the lyrics, such as "sing something"), the singing generation unit 522 generates a classification table. 5161 as the lyrics source.

In step S207, the song generation unit 522 requests the selected source to provide lyrics material. Here, an example is shown in which the classification table 5161, that is, the storage unit 516 is selected as the source. In this case, the request includes a user identifier and emotional information for that user. Upon receiving the request to provide lyric material, the storage unit 516 extracts the lyric material corresponding to the user identifier and emotion information included in the request from the classification table 5161 (step S208). The storage unit 516 outputs the extracted lyrics material to the singing generation unit 522 (step S209).

Upon acquiring the lyrics material, the song generation unit 522 requests the lyrics generation unit 5222 to generate lyrics (step S210). This request includes the lyric material obtained from the source. When generation of lyrics is requested, the lyrics generation unit 5222 generates lyrics using lyrics material (step S211). The lyrics generation unit 5222 generates lyrics by combining a plurality of lyrics materials, for example. Alternatively, each source may store lyrics for an entire song, and in this case, the lyrics generation unit 5222 selects lyrics for one song to be used for singing synthesis from among the lyrics stored in the source. You may. The lyrics generation unit 5222 outputs the generated lyrics to the song generation unit 522 (step S212).

In step S213, the song generation unit 522 requests the melody generation unit 5221 to generate a melody. This request includes user preference information and information specifying the number of notes in the lyrics. The information specifying the number of sounds in the lyrics is the number of characters, mora, or syllables of the generated lyrics. When generation of a melody is requested, the melody generation unit 5221 generates a melody according to the preference information included in the request (step S214). Specifically, for example, it is as follows. The melody generation unit 5221 is a database of melody materials (for example, a note string having a length of about 2 or 4 measures, or an information string in which a note string is subdivided into musical elements such as rhythm and pitch changes). (hereinafter referred to as "melody database", not shown). The melody database is stored in the storage unit 516, for example. The melody database records the attributes of the melody. The attributes of the melody include, for example, music information such as a suitable melody or lyrics, composer name, and the like. The melody generation unit 5221 selects one or more materials that match the preference information included in the request from among the materials recorded in the melody database, and combines the selected materials to create a melody of a desired length. obtain. The song generation unit 522 outputs information (for example, sequence data such as MIDI) that specifies the generated melody to the song generation unit 522 (step S215).

In step S216, the song generation unit 522 requests the melody generation unit 5221 to modify the melody or the lyrics generation unit 5222 to generate lyrics. One of the purposes of this modification is to match the number of notes in the lyrics (for example, the number of moras) with the number of notes in the melody. For example, if the number of moras in the lyrics is less than the number of sounds in the melody (if there are fewer characters), the singing generation unit 522 requests the lyrics generation unit 5222 to increase the number of characters in the lyrics. Alternatively, if the number of moras in the lyrics is greater than the number of notes in the melody (in case of excess characters), the singing generation unit 522 requests the melody generation unit 5221 to increase the number of notes in the melody. In this figure, an example of modifying lyrics will be explained. In step S217, the lyrics generation unit 5222 modifies the lyrics in response to the modification request. When modifying the melody, the melody generation unit 5221 modifies the melody by, for example, dividing the notes and increasing the number of notes. The lyrics generation unit 5222 or the melody generation unit 5221 may adjust the lyrics so that the section between the clauses of the lyrics matches the section between the phrases of the melody. The lyrics generation unit 5222 outputs the corrected lyrics to the song generation unit 522 (step S218).

Upon receiving the lyrics, the song generation unit 522 selects the segment database 5162 used for song synthesis (step S219). The segment database 5162 is selected, for example, depending on the user's attributes regarding the event that triggered the song synthesis. Alternatively, the segment database 5162 may be selected depending on the content of the event that triggered the song synthesis. Furthermore, the segment database 5162 may be selected according to user preference information recorded in the classification table 5161. The song generation unit 522 synthesizes the speech segments extracted from the selected segment database 5162 according to the lyrics and melody obtained in the previous processing, and obtains data of a synthesized song (step S220). Note that the classification table 5161 may record information indicating the user's preferences regarding singing rendition techniques such as changing the tone of voice in singing, tapping, shaking, and vibrato, and the singing generating unit 522 refers to this information. Then, a song that reflects a rendition style according to the user's preference may be synthesized. The song generation unit 522 outputs the generated synthetic song data to the synthesis unit 524 (step S2221).

Further, the singing generation unit 522 requests the accompaniment generation unit 523 to generate an accompaniment (S22
2). This request includes information indicating the melody in the singing synthesis. The accompaniment generation unit 523 generates accompaniment according to the melody included in the request (step S223). A well-known technique is used to automatically add accompaniment to a melody. If data indicating the chord progression of the melody (hereinafter referred to as "chord progression data") is recorded in the melody database, the accompaniment generation unit 523 may generate accompaniment using this chord progression data. Alternatively, if chord progression data for accompaniment to the melody is recorded in the melody database, the accompaniment generation unit 523 may generate accompaniment using this chord progression data. Alternatively, the accompaniment generation section 523 may store a plurality of pieces of accompaniment audio data in advance, and read out the accompaniment audio data that matches the chord progression of the melody. Further, the accompaniment generation unit 523 may refer to the classification table 5161 to determine the melody of the accompaniment, for example, and generate accompaniment according to the user's preference. The accompaniment generation unit 523 outputs the generated accompaniment data to the synthesis unit 524 (step S224).

Upon receiving the synthetic singing and accompaniment data, the synthesizing unit 524 synthesizes the synthetic singing and accompaniment (step S225). When compositing, the singing and accompaniment are synchronized by matching the starting position and tempo of the performance. In this way, data for synthetic singing with accompaniment is obtained. The synthesis unit 524 outputs data of synthesized singing.

Here, we have explained an example in which lyrics are first generated and then a melody is generated to match the lyrics. However, the voice response system 1 may first generate a melody and then generate lyrics in accordance with the melody. Furthermore, although an example has been described in which the singing and accompaniment are combined and then output, the accompaniment may not be generated and only the singing may be output (that is, a cappella may be used). Furthermore, here, an example has been described in which the accompaniment is generated in accordance with the singing after the singing is first synthesized, but the accompaniment may be generated first and then the singing may be synthesized in accordance with the accompaniment.

4. Response Function FIG. 12 is a diagram illustrating a functional configuration of the voice response system 1 related to the response function 53. As functional elements related to the response function 53, the voice response system 1 includes a voice analysis section 511, an emotion estimation section 512, and a content decomposition section 531. In the following, description of elements common to the learning function 51 and the singing synthesis function 52 will be omitted. The content decomposition unit 531 decomposes one content into a plurality of partial contents. In this example, content refers to the content of information output as a response voice, and specifically refers to, for example, songs, news, recipes, or teaching materials (sports lessons, musical instrument lessons, learning drills, quizzes).

FIG. 13 is a flowchart illustrating the operation of the voice response system 1 related to the response function 53. In step S31, the audio analysis unit 511 identifies content to be played. The content to be reproduced is specified, for example, according to the user's input voice. Specifically, the audio analysis unit 511 analyzes the input audio and identifies the content that is instructed to be played by the input audio. In one example, when an input voice saying "Tell me the hamburger recipe" is given, the speech analysis unit 11 instructs the processing unit 510 to provide the "hamburger recipe". The processing unit 510 accesses the content providing unit 60 and obtains text data that describes the “hamburger recipe.” The data thus obtained is specified as the content to be reproduced. The processing unit 510 notifies the content decomposition unit 531 of the specified content.

In step S32, the content decomposition unit 531 decomposes the content into a plurality of partial contents. In one example, a "hamburger recipe" is composed of multiple steps (cutting ingredients, mixing ingredients, shaping, baking, etc.), and the content decomposition unit 531 converts the text of "hamburger recipe" into "ingredients". It is broken down into four partial contents: ``cutting step,'' ``mixing ingredients,'' ``molding step,'' and ``baking step.'' The content decomposition position is automatically determined, for example, by AI. Alternatively, a marker indicating a break may be embedded in the content in advance, and the content may be decomposed at the position of the marker.

In step S33, the content decomposition unit 531 specifies one target partial content among the plurality of partial contents (an example of a specifying unit). The target partial content is the partial content to be reproduced, and is determined according to the positional relationship of the partial content in the original content. In the example of "hamburger recipe", the content decomposition unit 531 first specifies "step of cutting ingredients" as the target partial content. Next, when the process of step S33 is performed, the content decomposition unit 531 specifies the "step of mixing materials" as the target partial content. The content decomposition unit 531 notifies the content modification unit 532 of the identified partial content.

In step S34, the content modification unit 532 modifies the target partial content. The specific modification method is defined depending on the content. For example, the content modification unit 532 does not modify content such as news, weather information, and recipes. For example, with respect to the content of a teaching material or a quiz, the content modification unit 532 replaces a part that is desired to be hidden as a question with another sound (for example, humming, "la la la", beep sound, etc.). At this time, the content modification unit 532 replaces the character string with a character string that has the same number of moras or syllables as the character string before replacement. Content modification section 532 outputs the modified partial content to song generation section 522.

In step S35, the song generation unit 522 singly synthesizes the corrected partial content. The singing voice generated by the singing generation unit 522 is finally output from the input/output device 10 as a response voice. After outputting the response voice, the voice response system 1 enters a state of waiting for a user's response (step S36). In step S36, the voice response system 1 may output a song or a voice (for example, "Did you do it?") that prompts the user to respond. The voice analysis unit 511 determines the next process depending on the user's response. If a response requesting reproduction of the next partial content is input (S36: next), the audio analysis unit 511 moves the process to step S33. The response prompting reproduction of the next partial content is, for example, a voice such as "Go to next step", "Done", "Done", etc. If a response other than a response prompting reproduction of the next partial content is input (S36: End), the audio analysis unit 511 instructs the processing unit 510 to stop outputting audio.

In step S37, the processing unit 510 at least temporarily stops outputting the synthesized audio of the partial content. In step S38, the processing unit 510 performs processing according to the user's input voice. The processing in step S38 includes, for example, stopping reproduction of the current content, searching for a keyword instructed by the user, and starting reproduction of another content. For example, if a response such as "I want you to stop singing", "It's over", or "It's over" is input, the processing unit 510 stops playing the current content. For example, when a question-type response such as "How do you cut strips of paper?" or "What is Ario Olio?" is input, the processing unit 510 provides content with information to answer the user's question. 60. The processing unit 510 outputs the audio of the answer to the user's question. This answer may be spoken rather than sung. When a response instructing reproduction of another content, such as "play the song by XX", is input, the processing unit 510 acquires the instructed content from the content providing unit 60 and reproduces it.

Note that an example has been described here in which the content is decomposed into a plurality of partial contents and the next process is determined for each partial content according to the user's reaction. However, the method by which the response function 53 outputs the response voice is not limited to this. For example, content is not decomposed into partial content,
The content may be output as a spoken voice or as a singing voice using the content as lyrics. The voice response system 1 may determine whether to decompose into partial contents or output as is without decomposing, depending on the user's input voice or the content to be output.

5. Operation Examples Some specific operation examples will be explained below. Although not explicitly stated in each operation example, each operation example is based on at least one of the above learning function, singing synthesis function, and response function. Note that all the operation examples below are explained using Japanese, but the language used is not limited to Japanese, and any language may be used.

5-1. Operation example 1
FIG. 14 is a diagram showing an example 1 of operation of the voice response system 1. In this example, the user requests the reproduction of a song by inputting a voice that says, "Play 'Sakura Sakura' (song title) by Ichitaro Sato (performer name)." The voice response system 1 searches the music database according to this input voice and plays the requested music. At this time, the voice response system 1 updates the classification table using the user's emotion when inputting this input voice and the analysis result of this song. The classification table is updated every time a request is made to play a song. The classification table becomes more reflective of the user's preferences as the number of times the user requests the voice response system 1 to play music increases (that is, as the cumulative usage time of the voice response system 1 increases). go.

5-2. Operation example 2
FIG. 15 is a diagram showing a second example of operation of the voice response system 1. In this example, the user requests singing synthesis with the input voice "sing me some fun song." The voice response system 1 performs singing synthesis according to this input voice. When synthesizing a song, the voice response system 1 refers to the classification table. Lyrics and melodies are generated using the information recorded in the classification table. Therefore, it is possible to automatically create music that reflects the user's preferences.

5-3. Operation example 3
FIG. 16 is a diagram showing a third example of operation of the voice response system 1. In this example, the user requests the provision of weather information by inputting a voice that says, "What's the weather like today?" In this case, as a response to this request, the processing unit 510 accesses a server that provides weather information in the content providing unit 60 and obtains text indicating today's weather (for example, "Today is a clear day"). The processing unit 510 outputs a song synthesis request including the acquired text to the song generation unit 522. The song generation unit 522 performs song synthesis using the text included in this request as lyrics. The voice response system 1 outputs a singing voice of "Today is a sunny day" with a melody and accompaniment as a response to the input voice.

5-4. Operation example 4
FIG. 17 is a diagram showing a fourth example of operation of the voice response system 1. In this example, the user had been using voice response system 1 for two weeks and had often played romance songs before the illustrated response was initiated. Therefore, information indicating that the user likes love songs is recorded in the classification table. The voice response system 1 asks the user questions such as "Where is the best place to meet?" and "When is the best season?" in order to obtain information that can be a hint for lyrics generation. The voice response system 1 generates lyrics using the user's answers to these questions. In this example, since the period of use is still short, 2 weeks, the classification table of the voice response system 1 has not yet sufficiently reflected the user's preferences, and the correspondence with emotions has not been sufficiently established. Therefore, even though the user actually likes ballad-style songs, a different rock-style song is generated.

5-5. Operation example 5
FIG. 18 is a diagram showing operation example 5 of the voice response system 1. This example shows an example in which the voice response system 1 has been continued to be used since operation example 3, and the cumulative usage period is one and a half months. Compared to Operation Example 3, the classification table reflects the user's preferences more, and the synthesized singing matches the user's preferences. The user can experience that the response of the voice response system 1, which was initially incomplete, gradually changes to match his or her preference.

5-6. Operation example 6
FIG. 19 is a diagram showing operation example 6 of the voice response system 1. In this example, the user requests the provision of the content of "recipe" for "hamburger steak" by inputting a voice saying "Can you tell me the recipe for hamburger steak?" The voice response system 1 breaks down the content into partial content based on the fact that the content called "recipe" should proceed to the next step after a certain step is completed, and performs the next process according to the user's reaction. It is decided to play the data in a manner that determines.

The "recipe" for "hamburger" is broken down step by step, and each time the singing of each step is output, the voice response system 1 generates a voice prompting the user to respond, such as "Did you do it?", "Are you finished?" Output. When the user utters an input voice instructing the singing of the next step, such as "Done" or "What's next?", the voice response system 1 outputs the singing of the next step in response. When the user utters an input voice asking, "How do you chop an onion?", the voice response system 1 outputs a song "Chop an onion" in response. When the singing of "chopped onion" is finished, the voice response system 1 starts singing from the continuation of "recipe" for "hamburger steak".

The voice response system 1 may output the singing voice of another content between the singing voice of the first partial content and the singing voice of the second partial content that follows. For example, the voice response system 1 combines the singing voice of the first partial content with the singing voice of the first partial content and the singing voice of the second partial content, which has been synthesized to have a time length corresponding to the item indicated by the character string included in the first partial content. output between the singing voice and the singing voice. Specifically, if the first partial content indicates that a waiting time of 20 minutes will occur, such as "Let's simmer the ingredients for 20 minutes," the voice response system 1 will simmer the ingredients. Synthesize and output 20 minutes of singing while you are there.

In addition, the voice response system 1 outputs the singing voice synthesized using the second character string corresponding to the first character string included in the first partial content as the singing voice of the first partial content. Thereafter, the output may be output at a timing corresponding to a time length corresponding to the item indicated by the first character string. Specifically, if the first partial content indicates that a waiting time of 20 minutes will occur, such as "Let's simmer the ingredients for 20 minutes," the voice response system 1 will respond with "Simmering is completed." It is also possible to output the singing voice "It is" (an example of the second character string) 20 minutes after outputting the first partial content. Or, in an example where the first partial content is ``Let's simmer the ingredients for 20 minutes here'', when half of the waiting time (10 minutes) has passed, a message such as ``10 minutes left until the end of simmering'' is written. You may sing in the wind.

5-7. Operation example 7
FIG. 20 is a diagram showing operation example 7 of the voice response system 1. In this example, the user requests the provision of the content of the "memorization problem" of "world history" by inputting a voice saying, "Can you give me the memorization problem of the years in world history?" Based on the fact that the content "memorization questions" is for checking the user's memory, the voice response system 1 breaks down the content into partial content and determines the next process according to the user's reaction. Decide to play with.

For example, the voice response system 1 decomposes the era name memorization sentence "Thank you Himiko (239) Emperor of Wei" into two partial contents: "To Himiko" and "Thank you Emperor of Wei". do. The voice response system 1 outputs the song "Himiko ni" and waits for the user's response. When the user utters something, the voice response system 1 determines whether the voice uttered by the user is correct and outputs a voice according to the determination result. For example, when the user utters the correct voice, ``Thank you, Emperor of Wei,'' the voice response system 1 outputs a voice such as ``That's correct.'' Alternatively, if the user utters a voice that is not correct, such as "I don't understand," the voice response system 1 outputs the correct song, "Thank you, Himiko, Emperor of Wei."

5-8. Operation example 8
FIG. 21 is a diagram showing operation example 8 of the voice response system 1. Similar to operation example 7, the user requests the provision of the content of "memorization questions" for "world history." The voice response system 1 outputs a portion of the content "memorization questions" with a portion thereof hidden, based on the fact that the content is for checking the user's memory. The portion to be hidden may be defined in the content, for example, or may be determined by the processing unit 510, that is, the AI, based on the results of morphological analysis or the like.

For example, the voice response system 1 sings the era name memorization sentence "Thank you to Himiko (239) Emperor of Wei" while hiding the part of "Thank you to Himiko". Specifically, the voice response system 1 replaces the hidden portion with another sound or character string (for example, humming, "la la la", beep sound, etc.). The sound or character string used for replacement is a sound or character string that has the same number of moras or syllables as before replacement. In one example, the voice response system 1 outputs the song "Himiko la la la la la la Emperor of Wei." After outputting this singing, the voice response system 1 waits for the user's reaction. When the user utters some voice, the voice response system 1 determines whether the voice uttered by the user is correct and outputs a voice according to the determination result. For example, when the user utters a voice such as "Thank you Himiko, Emperor of Wei," the voice response system 1 outputs a voice such as "That's correct." Alternatively, if the user utters a voice saying "I don't understand," the voice response system 1 outputs the correct song, "Thank you Himiko, Emperor of Wei."

Furthermore, the voice response system 1 may replace part or all of the second partial content that follows the first partial content with another character string, depending on the user's reaction to the first partial content. For example, in the content of a question book or quiz, depending on whether the first question (an example of the first partial content) is answered correctly or incorrectly, the second question (an example of the second partial content) is answered differently. You may change the number of characters replaced in the string (for example, if the first question is correct, the second question hides more characters, and if the first question is incorrect, the second question is hidden). hides fewer characters).

5-9. Operation example 9
FIG. 22 is a diagram showing operation example 9 of the voice response system 1. In this example, the user requests the provision of the content of the "procedure manual" by inputting the input voice "Can you read me the procedure manual for the process at the factory?". Based on the fact that the content called "procedure manual" is for confirming the user's memory, the voice response system 1 breaks down the content into partial content and determines the next process according to the user's reaction. Decide to play with.

For example, the voice response system 1 divides a procedure manual at random positions and decomposes it into a plurality of partial contents. After outputting the singing of the first partial content, the voice response system 1 waits for the user's reaction. For example, for the content of the procedure "After pressing switch A, press switch B when the value of meter B becomes 10 or less", the voice response system 1 sings the part "after pressing switch A", Wait for user's reaction. When the user utters some voice, the voice response system 1 outputs the singing of the next partial content. Alternatively, at this time, the singing speed of the next partial content may be changed depending on whether the user can correctly say the next partial content. Specifically, if the user can say the next partial content correctly, the voice response system 1 increases the speed of singing the next partial content. Alternatively, if the user cannot say the next partial content correctly, the voice response system 1 reduces the singing speed of the next partial content.

5-10. Operation example 10
FIG. 23 is a diagram showing an operation example 10 of the voice response system 1. Operation example 10 is an operation example of dementia countermeasures for elderly people. In this example, the fact that the user is an elderly person has been set in advance through user registration or the like. The voice response system 1 starts singing an existing song, for example, in response to a user's instruction. The voice response system 1 temporarily stops singing at a random position or a predetermined position (for example, before the chorus). At that time, they send messages such as ``Hmm, I don't understand'' or ``I forgot,'' acting as if they have forgotten the lyrics. The voice response system 1 waits for the user's response in this state. When the user utters some voice, the voice response system 1 outputs a song continuing from the words, using (part of) the words uttered by the user as the correct lyrics. Note that when the user utters something, the voice response system 1 may output a response such as "Thank you". When a predetermined period of time has elapsed while waiting for a user's response, the voice response system 1 may output a voice such as "I remember" and resume singing from where it left off.

5-11. Operation example 11
FIG. 24 is a diagram showing an operation example 11 of the voice response system 1. In this example, the user requests singing synthesis with the input voice "sing me some fun song." The voice response system 1 performs singing synthesis according to this input voice. The segment database used for song synthesis is selected, for example, according to the character selected at the time of user registration (for example, if a male character is selected, a segment database of male singers is used). In the middle of a song, the user utters an input voice that instructs to change the fragment database, such as "change to a female voice." The voice response system 1 switches the segment database used for song synthesis according to the user's input voice. The segment database may be switched while the voice response system 1 is outputting the singing voice, or when the voice response system 1 is waiting for a user's response as in operation examples 7 to 10. May be done.

As already explained, the voice response system 1 may have a plurality of segment databases in which phonemes pronounced by a single singer (or speaker) in different singing styles or tones are recorded. In such a case, the voice response system 1 may combine, or add, a plurality of segments extracted from a plurality of segment databases at a certain ratio (utilization ratio) for a certain phoneme. Furthermore, the voice response system 1 may decide this usage ratio according to the user's reaction. Specifically, when two segment databases are recorded for a certain singer, one with a normal voice and one with a sweet voice, when the user utters the input voice "with a sweeter voice," the database with the sweet voice is created. By increasing the usage rate of the database of sweet voice fragments, and emitting the input voice ``Speak more and more sweetly'', the usage rate of the sweet voice segment database will be increased.

6. Modifications The present invention is not limited to the above-described embodiments, and various modifications are possible. Some modified examples will be explained below. Two or more of the following modifications may be used in combination.

In this paper, the singing voice refers to a voice that includes singing at least in part, and may include a portion that is only an accompaniment that does not include singing, or a portion that is only a speaking voice. For example, in an example in which content is decomposed into a plurality of partial contents, at least one partial content may not include singing. Singing may also include rapping or reciting poetry.

In the embodiment, an example has been described in which the learning function 51, the singing synthesis function 52, and the response function 53 are related to each other, but each of these functions may be provided independently. For example, the classification table obtained by the learning function 51 may be used, for example, to learn user preferences in a music distribution system that distributes music. Alternatively, the singing synthesis function 52 may perform singing synthesis using a classification table manually input by the user instead of the classification table generated by the learning function 51. Furthermore, at least some of the functional elements of the voice response system 1 may be omitted. For example, the voice response system 1 does not need to include the emotion estimation unit 512.

The assignment of functions to the input/output device 10, the response engine 20, and the singing synthesis engine 30 is not limited to that illustrated in the embodiment. For example, the voice analysis section 511 and the emotion estimation section 512 may be implemented in an input/output device. Moreover, the relative arrangement of the input/output device 10, the response engine 20, and the singing synthesis engine 30 is not limited to that illustrated in the embodiment. For example, the singing synthesis engine 30 may be placed between the input/output device 10 and the response engine 20, and may perform singing synthesis on a response that is determined to require singing synthesis among the responses output from the response engine 20. . Further, the content used in the voice response system 1 is not limited to that provided by the content providing unit 60, that is, that existing on a network or cloud. The content used in the voice response system 1 may be stored in a local device such as the input/output device 10 or a device capable of communicating with the input/output device 10.

The hardware configurations of the input/output device 10, response engine 20, and singing synthesis engine 30 are not limited to those illustrated in the embodiment. For example, the input/output device 10 may be a computer device with a touch screen and a display, such as a smartphone or a tablet terminal. In this regard, the user's input to the voice response system 1 is not limited to that via voice, but may also be input via a touch screen, keyboard, or pointing device. Further, the input/output device 10 may include a human sensor. In this case, the voice response system 1 may use this human sensor to control operations depending on whether the user is nearby or not. For example, when it is determined that the user is not near the input/output device 10, the voice response system 1 may perform an operation of not outputting voice (not returning dialogue). However, depending on the content of the voice output by the voice response system 1, the voice response system 1 may output the voice regardless of whether or not the user is near the input/output device 10. For example, as described in the second half of Operation Example 6, the voice response system 1 may output the voice indicating the remaining waiting time regardless of whether the user is near the input/output device 10 or not. Regarding detection of whether the user is near the input/output device 10, considering the case where the user does not move much, sensors other than the human sensor such as a camera or temperature sensor may be used, or multiple sensors may be used. may be used in combination.

The flowcharts and sequence charts illustrated in the embodiment are merely examples, and the operation of the voice response system 1 is not limited thereto. In the flowcharts or sequence charts illustrated in the embodiments, the order of the processes may be changed, some processes may be omitted, or new processes may be added.

The programs executed in the input/output device 10, the response engine 20, and the singing synthesis engine 30 may be provided in a state stored in a recording medium such as a CD-ROM or a semiconductor memory, or may be provided via a network such as the Internet. It may also be provided by download.

1... Voice response system, 10... Input/output device, 20... Response engine, 30... Singing synthesis engine, 51... Learning function, 52... Singing synthesis function, 53... Response function, 60... Content providing unit, 101... Microphone, 102 ...Input signal processing unit, 103...Output signal processing unit, 104...Speaker, 105...CPU, 106...Sensor, 107...Motor, 108...Network IF, 201...CPU, 202...Memory, 203...Storage, 204...Communication IF , 301...CPU, 302...Memory, 303...Storage, 304...Communication IF, 510...Processing section, 511...Speech analysis section, 512...Emotion estimation section, 513...Music analysis section, 514...Lyrics extraction section, 515...Preference Analysis unit, 516... Storage unit, 521... Detection unit, 522... Song generation unit, 523... Accompaniment generation unit, 524... Synthesis unit, 5221... Melody generation unit, 5222... Lyrics generation unit, 531... Content decomposition unit, 532... Content correction department

Claims (9)

a first acquisition means for acquiring an input voice indicating a user's request inputted via an input/output device;
a second acquisition means for acquiring a text indicating an answer to the request based on the input voice from the server;
The text is the lyrics, and the input voice of the user is input through the input/output device during a certain period in the past, and is generated from the input voice instructing to play a song from the input/output device. a singing synthesis means for synthesizing a singing voice using a melody according to preference information indicating the user's preferences regarding the characteristics, attributes, and lyrics of a song;
and output means for outputting the singing voice to the input/output device,
When the input voice acquired by the first acquisition means requests singing synthesis, the singing synthesis means uses a melody according to the preference information generated using the emotion estimated from the input voice. An information processing device that synthesizes the singing voice. The information processing apparatus according to claim 1, wherein the second acquisition means acquires, from the server, text of non-music information indicating a response to a request based on the input voice. The information processing apparatus according to claim 1 or 2, wherein the second acquisition means acquires a search result obtained using the input voice as a search key as the text. comprising a decomposition means for decomposing the text into a plurality of partial contents including a first partial content and a second partial content;
The information processing device according to claim 3, wherein the output means outputs the singing voice of the first partial content, waits for a reaction from the user, and then outputs the singing voice of the second partial content. The singing synthesis means synthesizes a singing voice of the first partial content, and then synthesizes a singing voice that prompts the user to respond to the first partial content,
The information processing device according to claim 4, wherein the output means outputs a singing voice that prompts a response from the user after outputting the singing voice of the first partial content. The information processing device according to any one of claims 1 to 5, wherein the singing synthesis means modifies one of the lyrics and the melody to match the other when synthesizing the singing voice. The information processing device according to claim 6, wherein the singing synthesis means performs the correction so that the number of sounds in the lyrics matches the number of sounds in the melody. Obtaining input audio indicating a user's request input via an input/output device,
obtaining text indicating an answer to the request based on the input audio from another server;
The text is the lyrics, and the input voice of the user is input through the input/output device during a certain period in the past, and is generated from the input voice instructing to play a song from the input/output device. Synthesize a singing voice using a melody according to preference information indicating the user's preferences regarding the characteristics, attributes, and lyrics of the song,
outputting the singing voice to the input/output device;
When the acquired input voice requests singing synthesis, in the synthesis of the singing voice, the singing voice is synthesized using a melody according to preference information generated using the emotion estimated from the input voice. are synthesized
How to output singing audio. to the computer,
Obtaining input audio indicating a user's request input via an input/output device,
obtaining text indicating an answer to the request based on the input audio from another server;
The text is the lyrics, and the input voice of the user is input through the input/output device during a certain period in the past, and is generated from the input voice instructing to play a song from the input/output device. Synthesize a singing voice using a melody according to preference information indicating the user's preferences regarding song characteristics, attributes, and lyrics,
outputting the singing voice to the input/output device;
When the acquired input voice requests singing synthesis, in the synthesis of the singing voice, the singing voice is synthesized using a melody according to preference information generated using the emotion estimated from the input voice. are synthesized
A program for executing processing.

Images