JP6691737B2 - Lyrics sound output device, lyrics sound output method, and program - Google Patents

Description

  The present invention relates to a method of outputting lyrics information along with reproduction of music.

  There is known a karaoke device that synthesizes and outputs lyrics data in advance of a karaoke song (for example, Patent Documents 1 and 2).

JP-A-4-67467 JP, 10-63274, A

  In the case of a karaoke device, since the reproduced music does not include lyrics, the lyrics sound output by the prior art does not become difficult to hear. However, if you are listening to normal music instead of karaoke, when you output the lyrics by the method of the prior art, the output lyrics sound overlaps the lyrics part included in the original music. It may be difficult to hear. Further, for example, when listening to music while driving a vehicle, the lyric voice output by the method of the prior art may be difficult to hear because it overlaps with a voice message or the like of a route guidance by the vehicle-mounted navigation device. ..

  Examples of the problems to be solved by the present invention include the above. It is an object of the present invention to provide a lyric voice for a user to easily listen to a lyric voice for singing the song while playing music including lyrics.

The invention according to claim 1 is a lyrics voice output device, wherein lyrics data acquisition means for obtaining lyrics data of a musical composition being reproduced, and lyrics voice data for generating lyrics voice data based on the lyrics data. Generating means, lyrics voice data shortening means for shortening the time length of the written lyrics voice data, and a time length by the lyrics voice data shortening means prior to the lyrics portion in the music being reproduced. Output means for outputting the shortened lyrics voice data so as to end at the beat position of the music piece .

According to an eleventh aspect of the present invention, there is provided a lyrics voice output method executed by a terminal device including a computer, wherein a lyrics data acquisition step of obtaining lyrics data of a music piece being reproduced, and a lyrics data acquisition step based on the lyrics data. The lyrics voice data generation step of generating the lyrics voice data, the lyrics voice data shortening step of shortening the time length of the lyrics voice data, and the lyrics voice prior to the lyrics portion in the music being reproduced. An output step of outputting the lyrics voice data whose time length has been shortened by the data shortening means so as to end at the beat position of the music .

A twelfth aspect of the present invention is a program executed by a terminal device equipped with a computer, wherein lyrics data acquisition means for acquiring lyrics data of a music piece being reproduced, and lyrics voice data based on the lyrics data. The lyrics voice data generating means for generating, the lyrics voice data shortening means for shortening the time length of the lyrics voice data, the lyrics voice data shortening means for temporally preceding the lyrics portion in the music being reproduced. The computer is made to function as an output unit that outputs the lyrics voice data of which the length is shortened so as to end at the beat position of the music .

It is a figure which shows the concept of assist vocal. It is a flow chart of assist vocal processing. It is a flow chart of speech information generation processing. An outline of the speech information generation process is shown. An example of making lyrics blocks will be shown. An example of a speech insertion method will be shown. An example of speech enhancement processing will be shown. The structure which concerns on the other example of a speech emphasis process is shown. The structure which concerns on the other example of a speech emphasis process is shown. It is a block diagram which shows the whole structure of a music reproduction system. It is a block diagram which shows the internal structural example of a terminal device. It is a flowchart of the assist vocal processing by the music reproduction system of the first embodiment. It is a flowchart of the assist vocal processing by the music reproduction system of the second embodiment. It is a flowchart of the assisted vocal process which reproduces only speech. It is a figure explaining the specific method of the music currently reproduced by the external source.

  In a preferred embodiment of the present invention, the lyrics voice output device includes an acquisition unit that acquires music specifying information that specifies a music being reproduced by an external device, and a reproduction position that determines a reproduction position of the music being reproduced. Determining means, lyrics data acquiring means for acquiring the lyrics data of the music being reproduced, lyrics voice data generating means for generating lyrics voice data based on the lyrics data, and based on the reproduction position, And an output unit for outputting the lyrics voice data prior to the lyrics portion in the music being reproduced.

  The above-mentioned lyrics voice output device specifies a music piece being played by an external device and determines a playback position of the music piece being played. Also, the lyrics data of the music being played is acquired, and the lyrics voice data is generated based on the lyrics data. Then, based on the reproduction position, the lyrics voice data is output prior to the lyrics portion in the music being reproduced. As a result, the user can listen to the lyric sound of the music played by the external device and sing along with the music.

  In an aspect of the above-mentioned lyrics voice output device, the acquisition unit includes a sound collection unit that collects the sound data of the music that is being reproduced, and a transmission unit that transmits the collected sound data to an external server. Receiving means for receiving the music specifying information of the music being reproduced, which is specified by the external server based on the collected voice data. In this mode, the audio data of the music being reproduced is transmitted to the server, and the music is specified by receiving the music specifying information.

  In another aspect of the above-mentioned lyrics voice output device, the receiving means is music reproduction position information indicating the elapsed time from the beginning of the music being reproduced of the audio data transmitted to the external server by the transmission means. From the external server, the reproduction position determining means determines the reproduction position based on the music reproduction position information and the elapsed time from the time when the transmitting means transmits the audio data to the external server. decide. In this aspect, the music reproduction position information indicating the elapsed time from the beginning of the music being reproduced is received from the server, and the current reproduction position is determined based on the information.

  Another aspect of the above-described lyrics voice output device includes interruption determination means for determining whether or not the reproduction of the music is interrupted, and the output means is configured to detect the lyrics voice when the reproduction of the music is interrupted. End data output. In this aspect, when it is determined that the reproduction of the music by the external device is interrupted, the output of the lyrics voice data is automatically ended.

  In another aspect of the above-mentioned lyrics voice output device, when the acquisition unit acquires the music identification information of a music different from the music that has been reproduced until then, the output unit outputs the lyrics voice data. To finish. In this aspect, when the music played by the external device is changed, the output of the lyrics voice data is automatically terminated.

  In another aspect of the above-mentioned lyrics voice output device, when the acquisition unit acquires the music identification information of a music different from the music that has been reproduced until then, the output unit changes the music to another music. The output of the corresponding lyrics voice data is continued. In this aspect, when the music played by the external device is changed, the lyrics voice data corresponding to the changed music is output.

  In another preferred embodiment of the present invention, a lyrics voice output method executed by a terminal device equipped with a computer comprises an acquisition step of acquiring music specifying information for specifying a music being reproduced by an external device, and the reproduction step. Reproduction position determining step for determining the reproduction position of the music being played, lyrics data acquiring step for acquiring the lyrics data of the music being played, and lyrics voice data generation for generating lyrics voice data based on the lyrics data And a step of outputting the lyrics voice data prior to the lyrics portion in the music being reproduced based on the reproduction position. Also by this method, the user can listen to the lyrics voice of the music being played by the external device and sing along with the music.

  In another preferred embodiment of the present invention, the program executed by the terminal device equipped with a computer is an acquisition unit for acquiring music specifying information for specifying a music being reproduced by an external device, Reproduction position determining means for determining a reproduction position, lyrics data acquisition means for acquiring lyrics data of the music being reproduced, lyrics voice data generation means for generating lyrics voice data based on the lyrics data, and a reproduction position Based on this, the computer is caused to function as an output means for outputting the lyrics voice data prior to the lyrics portion in the music being reproduced. By executing this program on a computer, the above terminal device can be realized. This program can be stored in a storage medium and handled.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[1] Assisted Vocal [1.1] Concept of Assisted Vocal When a user who is driving a vehicle plays and listens to music in the vehicle, he or she may want to sing the song being listened to. However, since the lyrics information cannot be seen while driving, the user cannot sing unless the lyrics of the song are stored.

  In the present embodiment, while the music including the lyrics is being reproduced, the lyrics included in the music are output as a voice signal to teach the user. Specifically, while playing a song stored in the memory of a terminal device, the lyrics included in the song are output as voice before the lyrics are played in the song. Tell the user. This allows the user to sing the song that is being reproduced even while driving. Also, users other than the driver can sing songs without looking at the lyrics collection.

  The function of outputting the contents of the lyrics by voice output to the user prior to the timing when the lyrics are reproduced in the music is called “assist vocal”. In addition, in the present embodiment, it is assumed that the reproduced music is not a karaoke but a normal music including lyrics.

  FIG. 1 shows the concept of assisted vocals. FIG. 1 schematically shows one music piece. The horizontal axis of FIG. 1 represents time. In one music piece, the lyrics portion is divided into a plurality of blocks. The part of the lyrics included in the played music is called "vocal". Also, in the music, a part other than vocal is called "interlude". Therefore, one music piece is usually composed of a plurality of interludes and a plurality of vocals.

  In the example of FIG. 1, the music is composed of three vocals 1 to 3 and a plurality of interludes. It is assumed that the content (lyrics) of vocal 1 is "aiueo", the content of vocal 2 is "kakikukeko", and the content of vocal 3 is "sashisuseso".

  In such a situation where the music is being reproduced, in the present embodiment, the lyrics "aiueo" corresponding to the vocal 1 is voice output prior to the timing when the vocal 1 in the music is reproduced. In this specification, the lyrics voice output by assisted vocal is referred to as "speech" to distinguish it from "vocal" included in the music.

  In the example of FIG. 1, the speech 1 corresponding to the vocal 1 is output prior to the vocal 1. Similarly, speech 2 is output prior to vocal 2 and speech 3 is output prior to vocal 3.

  Speech outputs only the vocal lyrics included in a song as an audio signal, and basically does not include elements such as pitch and rhythm. Also, as will be described later, since the speech is basically inserted in the interlude before the corresponding vocal, its length is adjusted as necessary, and usually, it is better than when it is played as vocal during the playback of music. Is also a short time. In a typical example, the speech is a spoken voice of the lyrics of the corresponding vocal.

[1.2] Assisted Vocal Processing Next, the assisted vocal processing for outputting speech will be described. FIG. 2 is a flowchart of the assist vocal processing. It should be noted that this process is executed by a terminal device mounted on the vehicle, typically a mobile terminal such as a smartphone, the details of which will be described later. In the following description, it is assumed that the terminal device executes the process.

  First, the terminal device determines whether or not the assist vocal is turned on (step S1). Here, turning on / off of the assist vocal may be performed manually by the user or automatically. In the case of manual operation, the user operates a predetermined button or the like to turn on the assist vocal when he / she wants to reproduce the speech with the assist vocal, and the terminal device detects this. On the other hand, in the case of performing automatically, the terminal device determines the user's voice by using, for example, a microphone, and automatically turns on the assist vocal when the user is singing or performing an action similar to singing. Set. The method of automatically setting the assist vocal will be described later.

  If the assist vocal is not set to ON (step S1: No), the process ends. On the other hand, when the assist vocal is set to ON (step S1: Yes), the terminal device identifies the music being played (step S2). In this case, the music played in the car is music stored in the terminal device by being downloaded from a server, music stored in a storage medium such as a memory of a CD or vehicle-mounted device, a radio, and the like. Including songs that are being played from. When the music stored in the terminal device is being reproduced, the terminal device can easily identify the music being reproduced. On the other hand, when the music stored in the storage medium such as a CD is being reproduced or when the music is being reproduced from the radio, the terminal device collects the music reproduced from the speaker in the vehicle with the microphone. Then, the audio data is transmitted to an external music search server. The music search server stores a large number of pieces of music data in a database and stores the information. The music search server identifies a music piece that matches the audio data received from the terminal device and indicates the music piece (for example, song name, artist name, etc. ("Music identification information") is transmitted to the terminal device. In this way, the terminal device acquires the music identification information of the music currently being reproduced.

  In this way, when the music being reproduced is specified, the terminal device executes the speech information generation process (step S3). FIG. 3 is a flowchart of the speech information generation process. Further, FIG. 4 shows an outline of the speech information generation processing.

  In FIG. 3, the terminal device acquires the lyrics data of the song specified in step S2 from an external server or the like (step S31). Here, the "lyric data" is information that defines what kind of lyrics are reproduced at what timing in the music, and specifically, lyrics text data indicating the lyrics included in the music, This information is associated with the reproduction time data indicating the reproduction time at which the lyrics are reproduced (elapsed time from the start time of the song).

  Next, the terminal device acquires the music analysis data (step S32). The music analysis data is information indicating musical characteristics such as beat position and bar position in the music, and is generated based on the audio data of the reproduced music. Specifically, the terminal device has a built-in music analysis application, collects music played from the speaker of the vehicle with a microphone to acquire audio data, and analyzes the audio data to determine the beat position. Acquire song analysis data such as. Instead of incorporating the music analysis application in the terminal device, the music analysis data may be acquired by using an external music analysis device or a server.

  Next, the terminal device creates lyrics blocks (step S33). Lyrics block formation is a process of dividing the lyrics text data included in the lyrics data acquired in step S31 into blocks, and one block corresponds to one speech. That is, the lyrics block formation is a process of dividing the lyrics text data into speech units.

  In the example of FIG. 4, the terminal device has acquired “aiueo kakiku kosashi suseso” as the lyrics text data, and the terminal device uses this as three blocks of “aiueo”, “kaki kukeko”, and “sashisuseso”. To generate block lyrics data.

  FIG. 5 shows an example of lyrics block formation. FIG. 5A shows the first method. In this method, one block is provided between the interludes included in the music. The "interlude" is a part of the music other than "vocal". Specifically, when the length It of a section other than vocal (non-vocal section) is longer than the predetermined length t1, the terminal device determines that section as an interlude.

  However, exceptionally, a plurality of blocks may be combined into one block in relation to the length of the interlude. As in the example shown in FIG. 5B, when the length It2 of the interlude 2 immediately before the length Vt3 of the vocal 3 is very short (It2 <α1 · Vt3; α1 is an arbitrary coefficient), It is difficult to output the vocal of vocal 3 during interlude 2. In such a case, if the length It1 of the preceding interlude 1 is longer than the predetermined length, the terminal device sets the vocal 2 and the vocal 3 as one block. As a result, the speeches corresponding to the vocals 2 and 3 are given in the interlude 1.

  The second method is shown in FIG. In this method, the terminal device determines each block based on the division included in the lyrics data. That is, when the lyrics text data included in the lyrics data includes delimiter information in advance, the terminal device can block the lyrics text data according to the delimiters.

  Next, the terminal device makes lyrics speech (step S34). The block lyric data obtained by the lyric block is only text data indicating lyrics, and the lyric speech is a process of converting the block lyric data into voice data. Specifically, the terminal device incorporates text-to-speech conversion (TTS: TextToSpeech) software, and converts each block lyrics data obtained in step S33 into speech data. As a result, as shown in FIG. 4, speech 1 to 3 which are voice data are generated from each block lyrics data. Note that TTS conversion by an external server or the like may be used instead of incorporating the TTS software in the terminal device.

  Next, the terminal device changes the speech length (step S35). The speech length change is a process of shortening the time length of each speech obtained by making the lyrics speech so that the speech can be reproduced in a short time. As described above, each speech is reproduced in the interlude preceding the corresponding vocal. However, since the time length of the interlude is limited, it is necessary to shorten the speech for reproduction. Therefore, the speech length is changed.

Basically, the reproduction time of each speech is shortened (the reproduction speed is increased) within a range that can be heard by humans. For example, if the temporal length of each speech obtained in step S34 (referred to as "original speech length") is "St" and the speech length conversion coefficient is "α2", the speech length is changed by the speech length change. The length "Stv" is
Stv = St · α2 (α2 <1.0) (1)
Given in. For example, if α2 = 0.7, each speech will be reproduced at the speed of 30% of the original speed by changing the speech length.

  Further, in addition to the batch change as described above, the reproduction time may be further shortened according to the duration of the interlude corresponding to each speech. In this case, even if the speech has the same number of characters or the words have the same lyrics, the reproduction time differs depending on the position in the music (the length of the preceding interlude).

  Next, the terminal device calculates the speech insertion timing (step S36). The terminal device inserts the speech corresponding to a certain vocal in advance of the reproduction timing of the vocal. In the example shown in FIG. 4, the speech 1 corresponding to the vocal 1 is inserted before the vocal reproduction timing. Similarly, the speech 2 corresponding to the vocal 2 is inserted before the reproduction timing of the vocal 2, and the speech 3 corresponding to the vocal 3 is inserted before the reproduction timing of the vocal 3.

  A specific example of a method for inserting speech is shown in FIG. FIG. 6 shows an example of the timing of inserting the speech 2 corresponding to the vocal 2.

  In method 1, the speech ends a certain time before the start timing of the corresponding vocal. Specifically, as shown in FIG. 6, the speech 2 is inserted so as to end before a certain time T2 from the reproduction start timing of the vocal 2. That is, the speech 2 ends a certain time T2 before the reproduction of the vocal 2 starts. In this case, the reproduction start timing of the speech 2 is determined according to the length of the speech 2. In the method 1, since a certain time is secured from the end of the reproduction of the speech to the reproduction of the corresponding vocal, the user can sing the vocal part with a margin.

  In method 2, the timing of ending the speech is matched with the beat position of the music. Specifically, in the example of FIG. 6, the speech 2 is inserted so as to end N beats before the reproduction start timing of the vocal 2 (N is an arbitrary integer; N = 1 in this example). In this case, the reproduction start timing of the speech 2 is determined according to the length of the speech 2. The beat position of the music is obtained from the music analysis data described above.

  In method 3, both the reproduction start timing and the reproduction end timing of the speech are matched with the beat position of the music. Specifically, in the example of FIG. 6, the reproduction start timing and the reproduction end timing of the speech 2 are both matched to the third beat of the four beats.

  When the end timing of the speech or both the start timing and the end timing of the speech are made to coincide with the beat position of the music as in the methods 2 and 3, the speech is linked to the music, so that the user can easily sing the music.

  As described above, the terminal device determines the speech insertion timing. Specifically, for each speech, the reproduction start timing and the reproduction end timing are defined by the elapsed time from the beginning of the music. The reproduction start timing and reproduction end timing of each speech are stored as a part of the speech information. That is, the speech information includes an audio signal corresponding to each speech (hereinafter, also referred to as “speech signal”) and a reproduction start timing / reproduction end timing of each speech.

  Next, the process returns to the main routine shown in FIG. 2, and the terminal device acquires the current reproduction position of the music being reproduced (step S4). Specifically, the terminal device acquires the current reproduction position by counting the elapsed time from the reproduction start time of the music being reproduced.

  Next, the terminal device performs a speech enhancement process (step S5). The speech emphasis process is a process for distinguishing vocals included in music and speech to make them easier to hear, and details thereof will be described later.

  Next, the terminal device reproduces the speech based on the reproduction start timing / reproduction end timing of each speech included in the speech information and the current reproduction position (step S6). Specifically, the speech reproduction is started at the speech reproduction start timing, and the speech reproduction is ended at the speech reproduction end timing. As a result, the corresponding speech is reproduced prior to the vocal in the music.

  Next, the terminal device determines whether or not to end the reproduction of the speech (step S7). The case where the reproduction of the speech should be ended includes the case where the speech information is lost, the reproduction of the music itself is ended, the case where the assist vocal is turned off by the operation of the user, and the like. If the reproduction of the speech should not be ended (step S7: No), the process returns to step S4 to continue the reproduction of the speech. On the other hand, when the reproduction of the speech should be ended (step S7: Yes), the assist vocal processing ends.

[1.3] Automatic Assist Vocal Setting Method Next, a method for automatically setting the assist vocal to ON in step S1 of the assist vocal processing shown in FIG. 2 will be described.

  As a basic method, the terminal device collects the voice uttered by the user with a microphone, and the user is singing (singing a song) in accordance with the music or performing an action similar to singing. If it is judged, the assist vocal is automatically turned on. For example, if it is determined as a result of analyzing voice data collected by a microphone that a humming song is being performed, a piece of song is singing, or a person is humming, the assist vocal is turned on. On the other hand, when the voice data is not singing but a conversation with a passenger, the assist vocal is not turned on. Even if the voice data includes a humming part, even if the majority is conversation, the assist vocal is not turned on.

  Note that whether or not the user's voice included in the voice data is a song can be determined based on the presence or absence of the rhythm or pitch included in the voice data. For example, if the rhythm is regular or there is a large change in pitch, it is judged to be singing, and if the rhythm is irregular, it is judged to be not singing (is a conversation) if there is a small change in pitch. be able to. Further, by using the above-mentioned music analysis application, it may be determined that it is a singing when a beat or a bar can be extracted from the voice data, and it may be determined that it is not a singing when it cannot be extracted. Further, by using the above-described music search server or music search function, it may be determined that the song is a song when the music can be identified from the audio data, and it may be determined that the song is not a song when the music cannot be identified.

  Further, the terminal device calculates the correlation between the collected voice data and the music being played, and when there is a certain value or more of the correlation, determines that the user is singing and turns on the assist vocal. Good. Further, when the terminal device has already acquired the lyrics data of the song being played, if the user sings when the correlation between the voice data collected by the microphone and the lyrics data is a certain value or more. You may judge. Further, based on the lyrics data, if the user's voice is output even at the interlude position of the song where the lyrics should not exist, it may be determined that it is a conversation.

  Alternatively, information on the rhythm collected by the microphone may be used. For example, if it is determined that the user is striking the steering wheel with his or her hand or finger in time with the rhythm of the music, or if the user is stepping on the floor with the foot to take the rhythm, the user performs an action that is similar to singing It may be determined that the assist vocal is turned on. In this case, the correlation between the rhythm collected by the microphone and the rhythm of the music being reproduced may be calculated, and the assist vocal may be turned on when the correlation is a certain value or more. In addition, even if the correlation with the rhythm of the music being played back is not calculated, if the rhythm collected by the microphone is a repetition of a certain rhythm, the assist vocal may be turned on. ..

  Furthermore, the assist vocal may be turned on when the user's state is photographed by a camera that photographs the inside of the vehicle and the user shakes his / her head along with the music. In addition, even if the camera that captures the inside of the vehicle detects whether or not there is a passenger in the passenger seat or rear seat, and whether or not the user is singing or talking is changed depending on the presence or absence of the passenger, Good.

  In the above example, when the user determines that he is singing, the example in which the assist vocal is turned on has been described. However, even if the user is singing, the user knows the lyrics and plays the assist vocal. If it is determined that there is no need to do so, the assist vocal need not be turned on. Specifically, for example, when the correlation between the collected voice data and the music being played is a certain value or more and the correlation with the lyrics data is a certain value or more, the user knows the lyrics. Therefore, the assisted vocal is not turned on even when singing.

  However, in this case, since the user may not understand the lyrics in the middle of the process, the user may be prepared to generate and output the speech information. After that, if the correlation between the collected voice data and the music being played is less than a certain value, or if the correlation with the lyrics data is less than a certain value, the user is determined not to know the lyrics. , Assist vocal is output.

  Further, in the above example, the method of automatically setting the assist vocals has been described, but the automatic turn-off setting of the assist vocals can also be performed. While the assist vocal is turned on, the user does not sing along with the song (does not sing) or acts that are similar to singing (singing humming, singing pieces, humming When it is determined that the assist vocal is not performed, the assist vocal may be automatically turned off. Similarly, if a conversation is detected, assist vocals may be turned off automatically, or if it is determined that the rhythm is not being taken or the user is not shaking his / her head to the music, Vocals may be turned off automatically.

  In addition, in the above example, it is explained that the assist vocal is automatically turned on or off based on whether the user is singing or performing an action similar to singing. The automatic ON setting or the automatic OFF setting may be performed based on the above configuration. For example, for a user who wants to sing only the chorus portion of the song, when playing the chorus portion of the song, the assist vocal is automatically set to on, and when the portion other than the chorus portion of the song is played, Assisted vocals may be automatically set to off. On the other hand, for users who know the chorus part and want to practice the part other than the chorus, when the part other than the chorus of the song is played, the assist vocal is automatically set to ON and the Assisted vocals may be automatically turned off when playing a portion.

[1.4] Speech Enhancement Processing Next, the speech enhancement processing executed in step S5 of the assist vocal processing shown in FIG. 2 will be described. The speech enhancement process is a method for the user to distinguish between speech and vocal to make it easier to hear, and the following several methods are shown.

[1.4.1] Processing when speech and vocal overlap each other Basically, it is preferable that the speech is reproduced during the interlude just before the corresponding vocal and does not overlap the vocal in time. For this reason, the above-described speech length changing process (step S35) is performed. However, depending on the length of the speech and the length of the interlude, the speech may not be reproduced during the interlude even if the speech length is shortened. That is, when the length of the speech is longer than the length of the interlude, the speech and the vocal are partially overlapped and reproduced. As described above, one of the following processes may be performed instead of reproducing the speech and the vocal in a superimposed manner.

  (1) Adjust the vocal level.

  If the speech and the vocal overlap, there is a method to lower the volume level of the vocal. FIG. 7A shows a case where the rear part of the speech and the head part of the vocal overlap and an overlapping part X occurs. In this case, the vocal volume is adjusted in the overlapping portion X. Specifically, the volume of the vocal is lowered to a level at which the speech can be heard, or set to zero. As a result, the reproduction of the speech is prioritized in the overlapping portion X, and the speech can be easily heard.

  On the contrary, FIG. 7B shows a case where the beginning portion of the speech overlaps with the rear portion of the previous vocal, and the overlapping portion X occurs. Also in this case, the vocal volume is adjusted in the overlapping portion X. Specifically, the volume of the vocal is lowered to a level at which the speech can be heard, or set to zero. Further, in the overlapping portion X, the volume level of the vocal may be faded out and the volume level may be gradually reduced, instead of suddenly lowering the volume level of the vocal. As a result, the reproduction of the speech is prioritized in the overlapping portion X, and the speech can be easily heard.

  Specifically, the above level adjustment may be performed by lowering the volume level of the vocal component when the vocal component and the musical component of the musical instrument are separated in the music signal. On the other hand, if the vocal part is combined with the performance part of the musical instrument and the volume of only the vocal cannot be adjusted, the volume level of the entire music signal may be lowered, or the general volume of the music signal may be reduced. Alternatively, the volume level may be lowered only for the component in the frequency band corresponding to vocal (human voice).

  (2) Adjust the speech level.

  If the speech and the vocal overlap, there is also a method of lowering the volume level of the speech. FIG. 7C shows a case where the rear part of the speech overlaps with the head part of the vocal, and the overlapping part X occurs. In this case, the volume of speech is adjusted in the overlapping portion X. Specifically, the volume of speech is reduced or set to zero. Instead of abruptly lowering the volume of the speech, the volume of the speech may be faded out and gradually reduced. In this case, the speech cannot be heard at the overlapping portion X, but generally when the user listens to a piece of music that he / she knows to some extent, although he / she does not remember all the lyrics, if the beginning portion of the lyrics is known, after that, Often you can remember the lyrics and sing. Therefore, as shown in FIG. 7C, if the head portion of the speech can be heard, it is often acceptable that the back portion of the speech becomes hard to hear. This method is effective in such cases.

[1.4.2] Processing of making speech and vocals heard from different directions Humans have the ability to distinguish sounds coming from different directions at the same time (so-called cocktail party effect). By utilizing this, a method that allows the user to distinguish between speech and vocal can be considered. It should be noted that this method is executed regardless of whether the speech and the vocal overlap in time.

(1) Method of adjusting phase with left and right speakers FIG. 8A shows a configuration in which the phase of speech output from the left and right speakers is inverted. The music signal of the left (L) channel is supplied to the adder 32, and the music signal of the right (R) channel is supplied to the adder 33. On the other hand, the speech signal is supplied to the adder 33 as it is, and the phase is inverted by the phase inverter 31 and supplied to the adder 32. The output of the adder 32 is supplied to the left speaker 30L, and the output of the adder 33 is supplied to the right speaker 30R.

  According to this configuration, the sound image of the music including the vocal is localized between the left and right speakers, while the sound image of the speech is localized around the user's ear, and the user can distinguish the speech from the vocal in the music. It will be easier. In the example of FIG. 8A, only the phase of the speech signal supplied to the left speaker 30L is inverted by the phase inverter 31, but only the phase of the speech signal supplied to the right speaker 30R is inverted. You may invert it. Further, if there is a constant phase difference between the speech signals supplied to the left and right speakers, the sound image position of the speech and the sound image position of the music can be made different, so that the speech signal supplied to one speaker is not always required. There is no need to invert (change by 180 °). That is, it suffices to give a constant phase difference between the speech signal supplied to one speaker and the speech signal supplied to the other speaker.

(2) Method of controlling sound image localization FIG. 8B shows a configuration in which a sound image of speech can be set at an arbitrary position. The music signal of the left (L) channel is supplied to the adder 32, and the music signal of the right (R) channel is supplied to the adder 33. On the other hand, the speech signal is supplied to the adders 32 and 33 via the sound image localization control calculation unit 34 and the crosstalk cancellation unit 35. The sound image localization control calculation unit 34 convolves the transfer function between the target speaker position and the listening position (user's position) into the speech signal, and the crosstalk canceling unit 35 defines the speaker outputting the music and the listening position. Performs processing to cancel the transfer function between. As a result, the sound image of the music can be localized between the left and right speakers 30L and 30R, and the sound image of the speech can be localized at the target speaker position, which makes it easier for the user to distinguish between the speech and the vocal.

(3) Method of using headrest speaker When a headrest speaker is mounted on the seat of the vehicle in addition to the speaker of the vehicle, music including vocals can be output from the speaker of the vehicle and speech can be output from the headrest speaker. .. An example of the configuration in this case is shown in FIG.

  The music signals of the left and right channels are supplied to the speakers 30L and 30R of the vehicle, respectively. Further, the speech signal is supplied as it is to the right headrest speaker 35R, and the phase thereof is inverted by the phase inverter 31 and supplied to the left headrest speaker 35L. Also in this case, since the phase difference is given to the speech signals supplied to the two headrest speakers 35L and 35R, the sound image of the speech is localized at a position different from the sound image of the music, and the user can recognize the speech and the vocal in the music. Will be easier to hear. In this example as well, as in the example of FIG. 8A, a constant phase difference is provided between the speech signal supplied to one headrest speaker and the speech signal supplied to the other headrest speaker. You can do it.

  When the headrest speaker is used, the headrest speaker in the passenger seat may be used instead of the headrest speaker in the driver's seat to reproduce the speech. Further, when the headrest speaker is installed in a plurality of seats of the vehicle, the necessity of speech reproduction may be selected and set for each seat. By doing so, it is possible to set that the speech is reproduced only from the headrest speaker of the seat of the passenger who wants to hear the speech and sing a song.

  Further, instead of giving the phase difference, the sound image localization control calculation unit 34 and the crosstalk cancellation unit 35 are used in the same manner as the processing described in FIG. It may be localized to. This makes it easier for the user to distinguish between speech and vocal.

[2] System Configuration Next, a configuration example of a music reproduction system that realizes the above assist vocal will be described.

[2.1] First Example In the first example, the assist vocal processing is mainly executed on the terminal device side. FIG. 10 shows the overall structure of the music reproducing system according to the first embodiment. In the music reproducing system according to the first embodiment, a plurality of vehicles 1, a content provider 2, and a gate server 3 can communicate with each other via a network 4. The plurality of vehicles 1 can communicate with the content server 2 and the gate server 3 via the network 4 by wireless communication.

  The content provider 2 is a server such as a music distributor, and provides music data, music metadata, lyrics data, and the like. The gate server 3 is a server that functions to realize the assisted vocal according to the present embodiment, and acquires the music data, metadata, lyrics data, etc. of necessary music from the content provider 2 and stores it in a database (not shown). ing.

  An example of the internal configuration of the vehicle 1 is shown in FIG. The vehicle 1 includes a terminal device 10, a music reproducing device 20, and a speaker 30.

  The terminal device 10 is typically a mobile terminal such as a smartphone, and includes a communication unit 11, a control unit 12, a storage unit 13, a microphone 14, and an operation unit 15. The communication unit 11 communicates with the gate server 3 via the network 4. The control unit 12 includes a CPU and controls the entire terminal device 10.

  The storage unit 13 is a memory such as a ROM and a RAM, stores a program for the control unit 12 to execute various processes, and also functions as a work memory. When the control unit 12 executes the program stored in the storage unit 13, the processing including the assist vocal processing is executed. Further, the storage unit 13 may store the music data of the music saved by the user.

  The microphone 14 collects voices such as music played in the vehicle, singing by the user, and conversation, and generates voice data. The operation unit 15 is typically a touch panel or the like, and receives an input of operation and selection by the user.

  The music reproducing device 20 is, for example, a car audio device, and includes an amplifier. The speaker 30 is a speaker mounted on the vehicle. The music reproducing device 20 reproduces the music from the speaker 30 based on the music data supplied from the terminal device 10.

  Another example of the internal configuration of the vehicle 1 is shown in FIG. In this example, the vehicle 1 includes a terminal device 10x. The terminal device 10x is a device having both the functions of the terminal device 10 such as the mobile terminal shown in FIG. 11A and the music reproducing device 20 such as a car audio. Like the terminal device 10, the terminal device 10x includes a communication unit 11, a control unit 12, a storage unit 13, a microphone 14, an operation unit 15, and a music reproduction unit 16 corresponding to the music reproduction device 20. The terminal device 10x is connected to the speaker 30 and reproduces the music from the speaker 30 based on the music data.

  Next, the assist vocal processing by the music reproduction system of the first embodiment will be described. FIG. 12 is a flowchart of the assist vocal processing according to the first embodiment. In this processing, the assist vocal processing is mainly executed by the terminal device 10 or 10x (hereinafter, simply referred to as "terminal device 10").

  First, the gate server 3 connects to the content provider 2 via the network 4, acquires song data and lyrics data for a plurality of songs, and stores them in an internal database (step S101).

  The terminal device 10 receives the designation of the music piece to be reproduced by the user operating the operation unit 15 (step S102), and transmits the music piece designation information designating the music piece to the gate server 3 (step S103). The gate server 3 acquires the music data and the lyrics data of the music corresponding to the received music designation information from the database and transmits the music data and the lyrics data to the terminal device 10 (step S104).

  Next, the terminal device 10 uses the received song data and lyrics data to perform the processes of steps S105 to S109. Here, the processes of steps S105 to S109 are similar to steps S3 to S7 in FIG.

  Thus, in the music reproduction system of the first embodiment, the terminal device 10 mounted on the vehicle 1 mainly executes the assist vocal processing.

  In the above example, the gate server 3 acquires the music data from the content provider in step S101, but when the music data is stored in the terminal device 10, the gate server 3 acquires the music data from the terminal device 10. You may get it. Further, when the music data is stored in the database in the gate server 3, the music data may be acquired from that.

[2.2] Second Embodiment In the second embodiment, a part of the assist vocal processing is executed on the gate server 3 side. The overall structure of the music reproducing system according to the second embodiment is similar to that of the first embodiment shown in FIG.

  Next, the assist vocal processing by the music reproducing system of the second embodiment will be described. FIG. 13 is a flowchart of the assist vocal processing according to the second embodiment. In this process, the gate server 3 generates speech information, further generates music data with speech, and transmits it to the terminal device 10. The terminal device 10 receives and reproduces the music data with speech. The details will be described below.

  First, the gate server 3 connects to the content provider 2 via the network 4, acquires song data and lyrics data for a plurality of songs, and saves them in an internal database (step S201). And the gate server 3 produces | generates speech information based on the acquired music data and lyrics data about each music (step S202). This speech information generation process is the same as step S3 in FIG. 2, and therefore its explanation is omitted.

  After generating the speech information, the gate server 3 adds speech to the music data to generate music data with speech (step S203). Specifically, the gate server 3 synthesizes the speech signal corresponding to each speech with the music data at the timing calculated by the process of step S36 of FIG. 3 based on the generated speech information to generate the music data with speech. And store it in the database. That is, the music data with speech is data in which speech is reproduced in addition to music by reproducing the music data with speech.

  The terminal device 10 receives the designation of the music piece to be reproduced by the user operating the operation unit 15 (step S204), and transmits the music piece designation information designating the music piece to the gate server 3 (step S205). The gate server 3 transmits the music data with speech of the music corresponding to the received music designation information to the terminal device 10 (step S206).

  Next, the terminal device 10 reproduces the received music data with speech (step S207). Thereby, the speech is reproduced at an appropriate timing during the reproduction of the music. Next, the terminal device 10 determines whether or not to end the reproduction of the music (step S208). When the reproduction is to be ended (step S208: Yes), such as when the music is reproduced to the end or when the user stops the reproduction (step S208: Yes), the terminal device 10 ends the reproduction. On the other hand, when the reproduction of the music should not be ended (step S208: No), the process returns to step S207 and the reproduction of the music data with speech is continued.

  Thus, in the music reproduction system of the second embodiment, the music data with speech is generated on the gate server 3 side and provided to the terminal device 10. The terminal device 10 can listen to the music including the speech by reproducing the received music data with the speech.

  In the above example, the gate server 3 acquires the music data from the content provider in step S201, but when the music data is stored in the terminal device 10, the gate server 3 acquires the music data from the terminal device 10. You may get it. Further, when the music data is stored in the database in the gate server 3, the music data may be acquired from that.

[3] Assisted Vocal for Reproducing Only Speech In the above-mentioned assisted vocal processing, a speech is added to the music being reproduced by the terminal device 10 and reproduced. However, it would be convenient if speech could be added to a music piece reproduced from a source other than the terminal device 10, for example, a radio in the car, a CD (hereinafter referred to as “external source”). In this case, the terminal device 10 basically has only to generate the speech information by the method described above and reproduce only the speech at the timing corresponding to the reproduction position of the music piece reproduced from the external source.

  FIG. 14 shows a flowchart of the assist vocal processing in this case. First, the terminal device 10 collects music that is being reproduced from an external source by the microphone 14 to acquire reproduced music data (step S151), and transmits this to the gate server 3 (step S152).

  The gate server 153 receives the reproduced music data from the terminal device 10 and specifies the corresponding music and its reproduction position (step S153). Specifically, the gate server 3 includes a music search unit having the function of the above-described music search server, specifies the music piece based on the reproduction music data, and reproduces the reproduction position corresponding to the portion of the reproduction music data. Specify. Then, the gate server 3 transmits the lyrics data and the reproduction position information to the terminal device 10 together with the music name and artist name of the specified music (step S154).

  The terminal device 10 uses the received lyrics data to generate speech information (step S155). The speech information is generated by the same method as described with reference to FIG. The terminal device 10 can acquire the music analysis data by analyzing the reproduced music data acquired by the microphone 14 (the process of step S32 in FIG. 3).

  Next, the terminal device 10 calculates the current reproduction position in the music based on the reproduction position information acquired from the gate server 3 (step S156). This method will be described later. Next, the terminal device 10 performs a speech enhancement process (step S157) and reproduces the speech at an appropriate timing according to the music being reproduced by the external source (step S158). As a result, the speech is reproduced according to the music being reproduced from the external source.

  Then, the terminal device 10 determines whether or not the reproduction of the speech should be ended (step S159), and if not ended, the process returns to step S156 to continue the processing. On the other hand, when the reproduction of the music from the external source is finished, the music being played is changed to another music, the speech to be played is lost, etc. S159: Yes), and the process ends.

  Next, with reference to FIG. 15, a method of identifying the current reproduction position of the music in step S156 will be described. The reproduced music data transmitted from the terminal device 10 to the gate server 3 is actually data of a plurality of audio frames. That is, the terminal device 10 collects the music being reproduced by the external source with the microphone 14 and sequentially transmits it to the gate server 3 as a plurality of audio frames.

  In the example of FIG. 15, the terminal device 10 has audio frames n, (n + 1), (n + 2) ,. ． ． Are sequentially transmitted to the gate server 3 as reproduced music data. At this time, the terminal device 10 stores the time when the reproduction music data is first transmitted, in the example of FIG. 15, the time when the audio frame n is transmitted (hereinafter, referred to as “reference time t0”).

  The music search unit of the gate server 3 refers to the information of many music pieces stored in the database, and specifies the music piece based on the plurality of received audio frames. In the example of FIG. 15, it is assumed that the music search unit of the gate server 3 has been able to identify the music piece based on the audio frames n to (n + 4). In this case, the gate server 3 uses the reproduction time (tn) from the beginning of the music of the audio frame n first received from the terminal device 10 as the reproduction position information in addition to the music name, artist name, etc. as the music position determination result. Send to device 10. That is, the reproduction position information transmitted from the gate server 3 to the terminal device 10 in step S154 of FIG. 14 is the elapsed time from the beginning of the song of the audio frame n first transmitted by the terminal device 10 to the gate server 3. Is becoming Therefore, in step S156, the terminal device 10 calculates the elapsed time Δt of the elapsed time from the reference time t0 to the present, which is stored in advance, and adds this to the reproduction time tn. That is, the reproduction time tn transmitted from the gate server 3 is the time from the beginning of the music piece to the audio frame n, and the elapsed time Δt is the time from the audio frame n to the present. Therefore, the current reproduction position (reproduction time) Tc is calculated by the following formula.

Tc = tn + Δt (2)
As described above, by providing the music search function in the gate server 3 and specifying the music and its reproduction position based on the reproduction music data, it is possible to reproduce the speech in accordance with the music reproduced from the external source. .. Further, instead of providing the music search function in the gate server 3, an external music search server may be used.

  Note that in step S159, the reproduction may be ended when one music is ended, but if another music is being reproduced after one music is ended, the process is continued. Good. That is, the speech reproduction may be continued while the transmission of the music reproduction data from the terminal device 10 to the gate server 3 is continued. As a result, even if the music played from the external source changes, it is possible to continue playing the speech following the change.

  In the above configuration, the control unit 12 is an example of the reproduction position determining unit, the lyrics data acquiring unit, the lyrics voice data generating unit, the song data with lyrics voice generating unit, and the interruption determining unit of the present invention, and the microphone 14 is a main unit. The communication unit 11 is an example of the sound collecting unit of the invention, and the communication unit 11 is an example of the transmitting unit and the receiving unit of the invention.

1 vehicle 2 content provider 3 gate server 4 network 10, 10x terminal device 12 control unit 13 storage unit 14 microphone 20 music playback device 30 speaker

Claims (13)

Lyrics data acquisition means for acquiring the lyrics data of the music being played,
Lyrics voice data generation means for generating lyrics voice data based on the lyrics data,
Lyrics voice data shortening means for reducing the time length of the lyrics voice data,
Outputting means for outputting the lyrics voice data whose time length has been shortened by the lyrics voice data shortening means so as to end at the beat position of the song prior to the lyrics portion in the music being reproduced. ,
A lyric voice output device comprising:   The lyrics voice according to claim 1, wherein the lyrics voice data corresponding to the lyrics portion is output so as to be finished before the start timing of the lyrics portion in the music being reproduced. Output device. The lyrics voice output device according to claim 1 or 2, wherein
Further comprising a voice recognition means for recognizing voice,
A lyrics voice output device, wherein the output means for outputting the lyrics voice data starts operation when it is determined that the user is singing. The lyrics voice output device according to claim 1 or 2, wherein
The vehicle further comprises sound collecting means for collecting sound in the vehicle,
A lyrics voice output device, wherein the output means for outputting the lyrics voice data starts operation when the user determines that he is in rhythm. The lyrics voice output device according to claim 1 or 2, wherein
Further equipped with a photographing means for photographing the inside of the car,
A lyrics voice output device, wherein an output means for outputting the lyrics voice data starts operating when it is determined that the user is moving the body in accordance with the music. The lyrics data acquisition means is
Sound data collecting means for collecting sound data of the music being reproduced,
Transmitting means for transmitting the collected voice data to an external server,
Receiving means for receiving the music specifying information of the music being reproduced, which is specified by the external server based on the collected audio data.
The lyrics voice output apparatus according to claim 1, further comprising: The receiving means receives, from the external server, music reproduction position information indicating the elapsed time from the beginning of the music being reproduced of the audio data transmitted to the external server by the transmitting means,
The reproduction position determination means determines the reproduction position based on the music reproduction position information and an elapsed time from the time when the transmission means transmits the audio data to the external server. 6. The lyrics voice output device according to 6. An interruption determination means for determining whether or not the reproduction of the music is interrupted,
8. The lyrics voice output device according to claim 1, wherein the output unit ends the output of the lyrics voice data when the reproduction of the music piece is interrupted.   The output means ends the output of the lyrics voice data when the lyrics data acquisition means acquires the music identification information of a music different from the music that has been reproduced until then. 9. The lyrics voice output device according to any one of items 8 to 8.   When the lyrics data acquisition means acquires the music identification information of a music different from the music that has been played so far, the output means continues to output the lyrics voice data corresponding to the different music. 10. The lyrics voice output device according to claim 1, wherein A lyrics voice output method executed by a terminal device including a computer,
A lyrics data acquisition process that acquires the lyrics data of the song being played,
A lyrics voice data generation step of generating lyrics voice data based on the lyrics data;
A lyrics voice data shortening step for shortening the time length of the lyrics voice data,
An output step of outputting the lyrics voice data whose time length has been shortened by the lyrics voice data shortening means so as to end at the beat position of the song prior to the lyrics portion in the song being reproduced. ,
A lyrics voice output method comprising: A program executed by a terminal device including a computer,
Lyrics data acquisition means for acquiring the lyrics data of the music being played,
Lyrics voice data generation means for generating lyrics voice data based on the lyrics data,
Lyrics voice data shortening means for shortening the time length of the lyrics voice data,
Outputting means for outputting the lyrics voice data whose time length has been shortened by the lyrics voice data shortening means so as to end at the beat position of the song prior to the lyrics portion in the music being reproduced,
A program that causes the computer to function as.   A storage medium storing the program according to claim 12.