JP2009244607A - Duet part singing generation system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system for generating ideal duet singing in which virtual duet with a desired duet song can be carried out, even when the duet partner desired by an arbitrary user, for example, an intimate friend or acquaintance is not present there, or irrelevant of his or her singing capability, and in which setting of a key and a tempo of reproduced singing voice can be changed, and moreover, he or she can sing with a correct sound pitch and timing like a professional singer, in the duet piece of karaoke. <P>SOLUTION: The system for synthesizing voice of duet part singing, on the basis of user's singing voice which is obtained beforehand, in a Karaoke machine, comprises: a user ID obtaining means; a user classified phoneme data extraction managing means; a partner designation means and a duet part singing creation means. The singing voice is synthesized by a phoneme data extracted from the singing voice, note data of the duet song and score data for voice synthesizing, which is created from lyrics data, and the duet part singing is created so that voice of the duet part singing can be output. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a system capable of generating a counterpart duet part song by speech synthesis when an arbitrary user sings a duet song in a karaoke apparatus.

  In the current online karaoke system, in general, most of the karaoke music performed by the karaoke apparatus is sung by a single singer, but sings alternately by two men and women, so-called "dueet". Duet songs are also included. When singing such a duet song, two male and female singers are required, but the partner singer who is a partner may not be obtained.

  In such a case, conventionally, in addition to the accompaniment track, a male singing voice (singing voice) signal track and a female singing voice (singing voice) signal data track are separately provided. When a female sings a singing track, a system has been proposed in which a male singing track is selected and the singing voice of a partner singer who is a partner can be obtained (see Patent Document 1).

Moreover, even if the duet partner desired by the user is not present to the karaoke user, the user can sing a favorite duet song with the singing voice recorded in advance by the duet partner. The applicant has proposed a karaoke system that can be introduced to (see Patent Document 2).
JP-A-5-109187 JP 2006-208961 A

  In the case of the technique disclosed in Patent Document 1, male and female audio signals are recorded on a plurality of channels so that they can be reproduced continuously. Therefore, when a man sings a duet song, by specifying a female partner, the male duet part is canceled and not played, and when the female duet part is reached, the female song is played. is there.

  In the case of a system configured in this way, it is necessary to be able to cope with both the case where a man sings a duet song and the case where a woman sings, so a male song track and a female song track together with an accompaniment track. Both of these are necessary, and the amount of data of the audio signal is large, so that a storage medium having a large storage capacity is required. And with this technology, a duet partner outputs a sound that has been acquired by a professional singer in advance, and indeed, even if there is no duet partner, a virtual duet (single duet) can be made, but the partner is a "professional singer" Limited. Therefore, a virtual duet cannot be made with a user desired by an arbitrary user, for example, a close friend or acquaintance.

In this regard, the technique disclosed in Patent Document 2 has a mechanism that allows a virtual duet with a user desired by an arbitrary user. This technique is to acquire a singing voice signal from a microphone input during performance of karaoke music and record it as singing voice data in a hard disk device. The singing voice data recorded in this way is managed by the duet partner management table, and the user can specify the singing voice of a desired duet partner by specifying with an icon displayed on the remote control device.
However, with this technique, it is a matter of course that a desired duet partner must be able to sing a duet song desired by any user. That is, in a duet song in which a desired duet partner cannot sing, a virtual duet cannot be performed with the desired duet partner, and the duet song is limited.
Furthermore, in the prior art, the recorded singing voice data is based on the actually sung voice, and any user's key, tempo and skill of the singing ability will be recorded as they are and reproduced. The duet partner's singing voice could hardly be required to harmonize with any user's own singing.

  By the way, recently, speech synthesis technology has advanced, and a singing synthesizer capable of sampling a person's utterance and reproducing a singing has become high performance, even though the singing is not originally performed. Specifically, for example, in Japanese Patent Application Laid-Open No. 2007-240564, speech unit data indicating various speech units is sampled and stored in a speech unit (phoneme) database, and a note input from a user is recorded. Based on song data including data and lyric data, a song synthesis score (score data) is generated by chronologically indicating the progression of the song with information specifying a plurality of speech segments, generation timings, and pitches used for song voice synthesis. The speech unit data corresponding to the speech unit specified by the singing synthesis score is read from the speech unit database, and singing speech is synthesized by performing predetermined pitch conversion and unit connection. Yes.

Therefore, the present invention makes use of this advanced speech synthesis technology, and in a duet song of karaoke, any user who wants a duet partner, for example, a close friend or acquaintance, and these users are absent. Regardless of whether or not the song can be sung, any duet song (but the target song) can be a virtual duet, and can be harmonized with the key and tempo selected by the user at the time of singing. It is an object to provide a system that can generate an ideal duet song in which the pitch and timing do not change like a singer.
In view of this problem, the present inventors have made a solution by the means described below. That is, the duet part singing generation system according to claim 1 of the present invention includes a note data and lyrics data corresponding to a plurality of duet parts, and synthesizes a duet part song of a duet song to be played based on a performance sequence. A user ID obtained by logging in at an arbitrary karaoke device to identify the user, and when this user sings an arbitrary piece of music, from the singing voice acquired by a predetermined recording means Phoneme data is extracted, and the user ID of the user is linked and managed in the phoneme data management table for each user. When any user selects a predetermined duet song, the user can accept the designation of the duet partner. When this designation is accepted, the user-specific phoneme is based on the user ID of the duet partner. The phoneme data is extracted from the data management table, and the singing voice is synthesized with the score data for voice synthesis created from the note data of the duet song and the lyrics data, and the duet part song can be output as a voice. Is to be generated.

  In addition, the duet part singing generation system according to claim 2 of the present invention is the use of the invention according to claim 1, extracting facial expression data in addition to phoneme data from the singing voice collected by a predetermined recording means, The user's user ID is linked and managed by the user-specific phoneme data management table. Further, based on the user ID, facial expression data is added to the phoneme data from the user-specific phoneme data management table. At the same time, the singing voice is synthesized with this, the score data for voice synthesis created from the note data of the duet music and the lyrics data, and the duet part singing is generated so that the voice can be output.

  According to the first aspect of the present invention, when a user sings an arbitrary piece of music, the phoneme data is extracted from the singing voice acquired by a predetermined recording means and managed for each user. The phonetic data of the duet partner specified by the user is extracted, and the singing voice is synthesized from the note data of the selected duet song and the score data for voice synthesis created from the lyrics data, and the duet part singing is voiced. Since it is generated so that it can be output, regardless of whether or not the duet partner can sing the duet song, any user wants a duet partner, for example, by singing any song by a close friend or acquaintance From the extracted phoneme data, virtual duets can be made on any duet song with these people. In addition, since there is no need to hold the singing voice of the male part or female part with a large data capacity as in the past, and since the phoneme data with a relatively small capacity is used, the data storage amount at the time of the output of duet music Furthermore, since sound is output from the synthesizer standard such as the MIDI (registered trademark) standard and its sound source, the key and tempo can be changed (harmonized), and the pitch and timing can be adjusted. Since it doesn't go mad, it has the effect of creating an ideal duet song.

  According to the invention described in claim 2 of the present invention, in addition to the effect of the invention described in claim 1 above, since facial expression data is added, individuality can be imparted to the reproduced singing voice, and the partner The singing voice of the karaoke device can be improved by the singer, and the amusement performance of the karaoke apparatus can be improved.

  Hereinafter, embodiments of the present invention will be described based on examples based on a communication karaoke device. However, the present invention is not limited to this, and the present invention is also applied to a stand-alone karaoke device to obtain the same effect. be able to.

  As shown in FIG. 1, a karaoke apparatus 1 embodying the present invention is mainly composed of a central control means 2 for controlling the operation of the entire apparatus and various devices connected thereto. That is, the central control means 2 includes a communication control means 13 for controlling communication with the karaoke host device, an operation control means 16 for processing an operation signal from the remote control device R to the karaoke device 1, A singing voice recording means 17 for collecting a user's singing voice, a partner specifying means 18 for specifying a duet partner, and a music display means 19 for displaying information related to music in a selectable manner are incorporated. In addition to comprehensively controlling them, the hard disk device 4, RAM 5, performance sound source 6, mixer 7, sound system 8, PCM decoder 8, MPEG decoder 11, synthesis circuit 12, and ID card reading / writing as user ID acquisition means A mechanism called the device 3 is incorporated.

The performance sound source 6 forms a musical tone signal in accordance with performance data input by the music sequencer processing in the sequencer 15 executed by the central control means 2. The formed musical sound signal is input to the mixer 7, which is a plurality of musical sound signals generated by the performance sound source 6, a karaoke microphone M, and a user's singing voice signal input via the A / D converter 9. Is mixed with an appropriate balance. The mixed digital audio signal is input to the sound system 8. The sound system 8 includes a power amplifier, converts an input digital signal into an analog signal and amplifies it, and emits a singing sound from the speaker S.
This A / D converter 9 is input from the karaoke microphone M when a music piece selected with the user ID attached at the time of music selection (of course, any music other than a duet music) is played. It is an electronic circuit that digitally converts the singing voice at the time of singing and sends it to the singing voice recording means 17. The singing voice data collected by this singing voice recording means 17 is stored as the singing voice data 23 managed in the user-specific singing voice management table T1 of the HDD 4 in association with the user ID attached at the time of the music selection. The Then, the user-specific phoneme data extraction management means 24 extracts at least phoneme data 25, more preferably facial expression data (not shown) from the singing voice data 23, and associates it with a user ID for a predetermined user. It is managed by another phoneme data management table T2.

  The hard disk device 4 stores a music database 19, performance data 20 and video data (not shown) managed by the music database 19, and for example, each video data of the video data is encoded in MPEG format. Playback processing is performed by background video playback means (not shown) executed by the central control means 2, read out and input to the MPEG decoder 11. The MPEG decoder 11 converts the input MPEG data into an NTSC signal and inputs it to the synthesis circuit 12. The synthesis circuit 12 synthesizes an OSD such as a lyrics telop and a scoring display on the video signal of the background video. The synthesized video signal is displayed on the display means. The performance data 20 of a predetermined music (not limited to a predetermined duet music) stores note data and lyric data (not shown), respectively, and appropriately uses a singing technique such as vibrato or intonation for the music by frequency or the like. Stores music singing method data. Specifically, it has a music table in which a music ID, a music title, and an artist ID (artist name) are associated, and musical note data (for example, MIDI (registration) of a predetermined data format managed by the music ID for each music. (Trademark) format note data) and lyric data (lyric telop data) are stored in synchronism with music data as file names. Then, for a predetermined duet song, score data 21 for singing voice synthesis is created based on the note data and lyrics data, and the score data is stored for each song. There are various timings for creating the score data. In this embodiment, the score data is created in advance and incorporated in the music data. However, the present invention is not limited to this, and for example, any duet music is registered in the reservation queue. At this time, score data of the music may be created.

  The sequencer 15 reads the performance data identified by the music code from the performance data 20 of the hard disk device 4. This is mainly composed of a music sequencer and a lyrics sequencer. The music sequencer reads track data such as performance data tracks and guide melody tracks in the performance data, and controls the performance sound source 6 with this data to play karaoke music. On the other hand, the lyrics sequencer is provided with character pattern creation means, reads the data of the lyrics track in the performance data, creates a lyrics telop image pattern based on this data, and outputs it to the synthesis circuit 12.

  The karaoke device 1 is accompanied by a multi-function karaoke remote control device R having a music search function. With this remote control device R, the user transmits a desired performance reservation command to the karaoke device 1 by an infrared signal. To do. When the operation control means 16 detects this operation signal, it performs a corresponding process. For example, when a music code is input by the remote control device R, the music code is reserved for performance in the reservation queue of the RAM 5 as a request for karaoke music.

  FIG. 2 shows a front perspective view of the appearance of the remote control device R in the present embodiment. The remote control device R includes an ID card reading / writing device equipped with a non-contact IC module, and is configured as a user ID acquisition means 3. Strictly speaking, the ID card reading / writing device is an interface of user ID acquisition means, and a LAN port is provided in a predetermined lower part on the back of the remote control device, and various data are transmitted via the LAN connection cable. The read information is compared with a user database (not shown) via a dedicated communication stand (not shown) that also serves as a charger. The user ID is acquired for the first time. Further, the present invention is not limited to the user ID acquisition means. For example, a reading / writing device for a mobile phone with an IC function in which an IC module is built in a mobile phone, a password input device, a fingerprint, a voice, etc. An apparatus having a biometric identification function may be used.

  In front of the remote control device R, a user interface mainly including a display screen D2 by a liquid crystal display with a touch sensor is provided, and a GUI (Graphical User Interface) environment is provided. A search function is realized by searching a database (not shown) and displaying the result. The GUI also uses the GUI to select a duet song and specify a duet partner, which are necessary for the present invention.

  FIG. 3 is a display screen D2 of a login / logout instruction for each user using a user interface by an LCD attached to the remote control device R. On this display screen D2, an icon row 30 with a face photo for each user, a “login” icon 31 for logging in to the karaoke system, a “logout” icon 32 for logging out, and the like are displayed. In a state where the remote controller R is mounted on the above-described dedicated mount, first, the “login” icon 31 is selected, and then the ID card is attached to the ID card reading / writing device by loading the ID card. When the user ID is identified, login to the system is recognized.

  When the login is recognized, the user's face photo registered in advance in the karaoke host device, or if not registered in advance, a user's face photo taken using the digital camera function is newly registered and sequentially Then, an interface for each user is created in which an icon with a face photo for each user is posted for each user. When the icon 30 with a user-specific face photo is formed in the icon row 30 with a user-specific face photo, each icon is regarded as a currently logged-in user. For example, in FIG. Information about duet songs can be displayed to the currently logged-in user so that the user can select a song. When any user designates a duet partner of a predetermined duet song, the partner designating means 18 selects the desired user-specific photo icon 30a as described below. The partner can be identified.

  Next, the interface display screen D2 for designating a duet partner shown in FIG. 4 will be described below. The user can access this display screen D2 by a predetermined operation after selecting the icon 30a with a user-specific face photo in FIG. This display screen D2 is provided with various icons for associating the user ID with respect to a duet song selected by an arbitrary user and designating a desired duet partner. When the icon 30a with a user-specific face photo in FIG. 3 is selected, for example, the user field 33 displays the name and nickname of the user registered in advance together with the face photo 33a for identifying the user. Is done.

  As shown in FIG. 4, the user specified in this way is first selected by inputting a music code into the music code input section of the desired duet music designation field 34, and further, the duet music is selected as a duet partner. A duet part (male part or female part) to be sung is designated by selecting the duet part type field 34b. In the present embodiment, the duet part of the opposite sex for the user is automatically set by default from the gender of the attribute data of the specified user.

  The duet partner is designated by inputting into the duet partner designation field 35. In this embodiment, when a predetermined ID number written on the duet partner's ID card or the like is entered, a face photograph of the corresponding ID. The name and nickname 35b are displayed together with 35a, and when the user designates the icon by the GUI function and selects the “designation” icon 36 in a state where the duet partner is decided, the partner designation means 18 in FIG. The user ID and the music ID are associated with each other and stored as duet partner data 5a.

  Next, the user-specific phoneme data management table T2 will be described. As shown in FIG. 5, this user-specific phoneme data management table T2 corresponds to the user-specific singing voice management table T1, and a “number” field f1 for specifying each record in common, “User ID” field f2 for specifying, “Song ID” field f3 for specifying the song sung by the user, “Singing voice data” for specifying the singing voice data itself or the singing voice data The field f4, the phoneme data itself extracted from the singing voice data or the “phoneme data” field f5 for specifying the phoneme data, and the facial expression data itself or facial expression data extracted from the singing voice data in the same manner Each data is managed as an “expression data” field f6 for specifying.

  Hereinafter, the user-specific phoneme data extraction management means will be described in further detail. Phoneme data is extracted from singing voice data and generated for each user ID. Phoneme data is an element required for current speech synthesis. Individual speech sounds are divided into five speech segments: vowels, head consonants, tail consonants, changes from consonants to vowels, and changes from vowels to consonants. It is classified into data. In the present invention, for example, a singing voice processing device is provided, and a specific configuration thereof is shown in FIG. The singing voice processing device 40 includes a facial expression database 42 together with a phoneme database 41. The phoneme database 41 analyzes a voice signal of the singing voice data using an analysis method such as SMS analysis means or LPC analysis means, A speech segment carved for each user ID is stored for each user ID.

  The facial expression database 42 stores vibrato information that is an important element for singing facial expressions. This facial expression data is a singing technique that characterizes a song such as vibrato or intonation using frequency information and the like, and the singing technique can be reflected in the singing voice. In particular, the vibrato information is an important element for determining the individuality of the singing sound to be played, and by adjusting the parameters, for example, singer information (sex / child / young / middle-aged, etc.) Pops tone, Enka tone, etc.) can be specified arbitrarily. Therefore, it is possible to generate various singing facial expressions by giving arbitrary parameters to the facial expression data.

  Next, the speech segment selection unit 43 receives performance data (for example, MIDI information including information such as notes, lyrics, pitch bends, dynamics, etc.) in units of frames (frame data), and in the input frame data. Has a function of selecting and reading out speech unit data corresponding to the lyric data from the phoneme database 41. In this case, when a user ID is designated, speech segment data corresponding to the user ID is obtained.

  The facial expression template selection unit 44 reads specific vibrato data corresponding thereto from the facial expression database 42 according to various designations (for example, female / enka etc.) by the user from the setting unit 45. The setting unit 45 receives performance data (MIDI) and setting data from the remote controller R or the like. As a result, the vibrato information is selected by the facial expression template selection unit 44 based on the setting data, and the speech unit is selected by the speech unit selection unit 43. In the selection of the speech unit in the speech phoneme selection unit 43, when MIDI data is adopted as the performance data, the speech unit corresponding to the lyric information assigned with the syllable set in advance in the MIDI standard note data is selected as the phoneme. Read from the database 41.

  The speech segment and the vibrato information determined in this way are input to the control unit 46 according to the performance sequence. The singing voice sound source 47 controlled by the control unit 46 holds timbre information of a human voice for generating a voice signal from MIDI data, and generates a voice signal of the singing sound according to the control of the control unit 45. To do. The audio processing unit 48 holds various types of information for performing audio processing (such as reverb / chorus / variation) on the audio signal, and performs various types of audio processing on the audio signal under the control of the control unit 46.

  Since the singing voice processing device 40 is configured as described above, when performance data is input in accordance with the performance sequence, the voice segments are sequentially read based on the lyrics information, and the voice processing is performed based on the set vibrato information. The desired synthesized singing voice CV can be obtained. Moreover, since the setting content of the setting part 45 can be changed easily, the key and tempo of the synthetic singing voice CV to be output can be freely changed. Then, the output synthesized singing voice CV is further synthesized with the output from the performance sound source 6 in the mixer 7, and the synthesized singing voice CV and the performance sound of the karaoke music are obtained.

  When the singing voice processing device 40 is caused to function in this way, the synthesized singing voice CV is also reproduced at the same time as the performance sound. Therefore, when functioning as an original karaoke device, the setting data for stopping the function of the singing voice processing device 40 is given to the setting unit 45, so that the synthesized singing voice CV is not output and only the performance sound from the performance sound source 6 is output. Karaoke performance is possible.

  The present invention makes effective use of such a function. By giving data specifying a male or female duet part to the setting data to the setting unit 45 based on an instruction from the remote control device R, the duet music of the karaoke music is provided. When a man sings, a female duet part can be output as a synthesized singing voice CV, and when a woman sings, a male duet part can be output as a synthesized singing voice CV.

  As described above, according to the present invention, since the singing voice of the duet partner in the duet song can be obtained by the synthesized singing voice, it is possible to match the key and pitch of the singer who actually sings, The timbre of the synthesized singing voice can be freely selected by arbitrarily selecting a template. Regardless of whether or not the duet partner can sing the duet song, the duet partner desired by any user, for example, from phoneme data extracted by singing any song by a close friend or acquaintance This makes it possible to improve entertainment such as enabling a virtual duet on any duet song.

  According to the present invention, since the synthesized singing voice can be generated from the singing voice data, it is not necessary to record and store a huge amount of singing data for each duet song, and various arrangements can be made by software processing. As a result, the functionality of the karaoke apparatus can be remarkably improved.

The block diagram explaining the outline | summary of this invention. Front perspective view of remote control device Interface display screen for user login / logout instructions. Interface display screen for specifying duet partners. The conceptual block diagram of the phoneme data management table classified by user. The block diagram which shows the structure of a song voice processing apparatus.

Explanation of symbols

1 ... Karaoke device 2 ... Control means 3 ... ID card reading / writing device (user ID acquisition means)
4 .... Hard disk drive 5 .... RAM
5a... Designated partner data 17... Singing voice recording means 18... Partner designation means 21... Score data 24. 40 .. Singing voice processing device 41... Phoneme database 42 .. Expression database 43... Speech element selection unit 44 .. Expression template selection unit 45. ... Setting part 46 ... Control part 47 ... Singing sound source 48 ... Sound processing part M ... Karaoke microphone CV ... Synthetic singing voice T1 …… Singing voice management table by user T2 …… Phoneme data management table by user

Claims (2)

In a karaoke device comprising note data and lyrics data corresponding to a plurality of duet parts, and having a duet song to be played based on a performance sequence, the duet part singing based on the user's singing voice acquired in advance A user ID acquisition unit, a user-specific phoneme data extraction management unit, a partner designation unit, and a duet part song generation unit,
(A) The user ID acquisition means is to acquire a user ID logged in with an arbitrary karaoke device and identify the user,
(B) The user-specific phoneme data extraction management means extracts the phoneme data from the singing voice acquired by the predetermined recording means when the specified user sings an arbitrary piece of music. User IDs are linked and managed in the phoneme data management table for each user.
(C) Partner designation means that any user can accept a duet partner designation by the user when selecting a specific duet song.
(D) Duet part singing generation means, when receiving the designation of the duet partner, based on the user ID of the duet partner, extracts the phoneme data from the user-specific phoneme data management table and Singing voice is synthesized from the data, score data for voice synthesis created from the note data and lyrics data of the duet song, and the duet part song is generated so that voice output is possible.
This is a duet part singing system. The user-specific phoneme data extraction management means extracts facial expression data in addition to the phoneme data from the singing voice collected by the predetermined recording means, and associates the user ID of the user with the user-specific phoneme data management. The duet part singing generation means extracts facial expression data in addition to the phoneme data from the user-specific phoneme data management table based on the user ID of the duet partner, and also manages the phoneme. Singing voice is synthesized from the data and facial expression data, and the score data for voice synthesis created from the note data and lyrics data of the duet song, and the duet part song is generated so that voice output is possible.
The duet part singing system according to claim 1.

Images