JP4612329B2 - Information processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the failure that a conventional type automatic performance device, etc., do not assume that users perform practice of voice imitation in an information processor, etc, with which users can perform practice of voice imitation, etc. <P>SOLUTION: This information processor is equipped with; a voice data storage part 101 in which voice data is stored; a voice acquiring part 102 for acquiring voices; a first featured amount extracting part 103 for extracting prescribed featured amounts of voices acquired by the voice acquiring part; a second featured amount extracting part 104 for extracting prescribed featured amounts from the voice data stored in the voice data storage part; a comparison part 105 for comparing the featured amounts extracted by the first featured amount extracting part with the featured amounts extracted by the second featured amount extracting part; and an output part 106 for outputting results compared by the comparison part. By such constitution, users can perform practice of voice imitation easily. <P>COPYRIGHT: (C)2006,JPO&amp;NCIPI

Description

  The present invention relates to an information processing apparatus that can practice voice imitation and the like.

One's own voice that can be heard when a person speaks is a synthesized voice of one's voice (air conduction feedback) heard through the air and one's voice (bone conduction feedback) heard through the skull. Since the voices heard by others are only sounds that travel in the air, it is impossible to let others hear the voices they hear. This is why my voice or recorded voice heard through the microphone feels strange and strange. When simulating a voice at karaoke or a banquet, you thought that you were very similar to yourself, but the voice through the microphone is not as similar as you think, and the audience is bad There is also.
From this background, those who have noticed imitation of one person, those who want to learn one shot, those who are in trouble by being forced to perform one at a banquet, those who want to improve their voice imitation The technology to support is necessary.

  As a technique related to the technique for solving the above problems, there are the following automatic performance apparatuses. The automatic performance apparatus is an automatic performance apparatus that generates a musical sound based on previously stored musical sound data and performs an automatic performance. The automatic performance apparatus generates a musical sound by converting means for inputting voice and converting it into a singing signal. A signal generating means for generating a trigger signal at a predetermined interval, a first counting means for counting the number of times the trigger signal is generated by the signal generating means, and when the trigger signal is generated by the signal generating means, Evaluation is performed according to a ratio of the number counted by the second counting means to the number counted by the first counting means, and second counting means for counting the presence or absence of the singing signal converted by the converting means. There is an automatic performance device that includes a calculation unit that calculates a result and a notification unit that notifies an evaluation result calculated by the calculation unit (see Patent Document 1). This automatic performance device relates to an automatic performance device with a singing ability evaluation function suitable for intellectual education and the like that can evaluate a voice sung according to an accompaniment, and a karaoke function capable of singing a song on an accompaniment sound and its It aims at providing the automatic performance apparatus with a singing power evaluation function suitable for the intellectual education of the infant which has a singing power evaluation function and the actual feeling that singing power is evaluated is obtained.

Moreover, there is Primavista (registered trademark) as a music software product in which related technology is introduced (see Non-Patent Document 1). This software is a choral practice software with four functions: “pitch graph”, “sound taking mode”, “speech training”, and “humor measurement”. The function of the “pitch graph” is a function that displays a change in pitch in a graph when singing into a PC microphone. This function allows you to practice accurate pitches. The “sound taking mode” function is a function for practicing the choral part. When you sing while listening to the sound of other parts or your own part, the pitch of the sound is displayed on the score. The function of “speech training” is a function that displays musical notes and musical pitch problems as a musical score, and can practice notation and musical pitch by singing the musical score. The function of “Horimori measurement” is a practice function of the hamori, and when the harmony is applied to the sound from the PC or when two people harmonize, the pitch of the chord is displayed.
JP-A-5-11687 (first page, FIG. 1 etc.) Kawai Gakki Mfg. Co., Ltd. website, Internet <URL: http://www.kawai.co.jp/cmusic/products/primavista.htm>

However, the above-described conventional technology does not assume practice of voice imitation. In other words, in the prior art, it is considered whether a person who has heard the voice uttered to resemble something feels similar when the similarity of the feature quantity of the voice data is high. Not.
Therefore, in the prior art, it is possible to evaluate the singing ability and practice singing so that the pitch does not go wrong, but it is determined whether the voice uttered is similar to the target voice in accordance with the human sense. It was difficult.
In addition, according to the prior art, it has not been possible to practice such as performing arts so that only a part of the stored voice is similar. This is a trick that is said to be a one-off performance.
In addition, in the prior art, it is not possible to display the degree of similarity in real time with an index that is close to the index that a person feels. For example, when imitating a song, the trajectory cannot be corrected midway. It was.

  Furthermore, in the prior art, for example, the voice data of the correct song was forcibly changed to sound data and the similarity to the sound data could not be output, so practice to sing a song forcibly I could not. In addition, if you can sing a song forcibly, it will be useful as a banquet art.

The information processing apparatus according to the first aspect of the present invention includes a sound acquisition unit that acquires sound, a first feature amount extraction unit that extracts a predetermined feature amount of the sound acquired by the sound acquisition unit, and the first feature amount extraction An information processing apparatus comprising: a comparison unit that compares a feature amount extracted by the unit with a feature amount of comparison audio data; and an output unit that outputs a result of comparison by the comparison unit. Note that the predetermined feature amount preferably includes one or more information among information on vibrato extracted from audio data, information on how to enter sound, and information on change in pitch.
With this configuration, voice imitation can be practiced easily. In addition, the ability to imitate voices that people feel similar to can be obtained.

Further, the information processing apparatus of the second invention is capable of dividing the audio data into predetermined parts with respect to the information processing apparatus of the first invention, and the comparing unit is provided for each of the parts, The feature amount extracted by the first feature amount extraction unit is compared with the feature amount of comparison audio data, and the output unit is an information processing apparatus that outputs a comparison result for each portion output by the comparison unit. .
With this configuration, it is easy to practice part of voice imitation.

An information processing apparatus according to a third aspect of the present invention is an information receiving apparatus according to the second aspect of the present invention, in which an input receiving unit that receives an input for instructing the part and an input receiving unit that receives the input. The information processing apparatus further includes an audio output unit that reads out a part of audio data corresponding to the instructed part and outputs the audio.
With this configuration, it is easy to practice part of voice imitation.

An information processing apparatus according to a fourth aspect of the present invention is an information receiving apparatus according to the second aspect of the present invention, wherein an input receiving unit that receives an input that indicates the part and an input that the input receiving unit indicates the part are received. The voice acquisition unit acquires the voice, the first feature amount extraction unit extracts a predetermined feature amount of the voice acquired by the voice acquisition unit, and the comparison unit includes the first feature amount. The feature amount extracted by the extraction unit is compared with the feature amount of the comparison audio data, and the output unit is an information processing apparatus that outputs a result of comparison by the comparison unit.
With this configuration, it is easy to practice part of voice imitation.
Moreover, the information processing apparatus of 5th invention is an information processing apparatus with which the said output part outputs visually the result which the said comparison part compared with said information processing apparatus.
With this configuration, the voice imitation index is obvious and it is easy for the user to practice voice imitation.

Furthermore, an information processing apparatus according to a sixth aspect of the present invention is a sound that receives an input of sound shift information that is information indicating a degree of change of audio data stored in the audio data storage unit with respect to the information processing apparatus. The information processing apparatus further includes a deviation information input receiving unit and a voice data changing unit that changes the voice data based on the sound gap information.
With such a configuration, for example, it can be practiced to forcibly change the voice data of a correct song to voice data that is forcibly and forcibly sing a song.
Note that the above information processing apparatus may be realized by software.

  According to the present invention, voice imitation etc. can be practiced.

Hereinafter, embodiments of an information processing apparatus and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, description may be abbreviate | omitted again.
(Embodiment 1)

FIG. 1 is a block diagram of an information processing apparatus according to this embodiment. The information processing apparatus includes an audio data storage unit 101, an audio acquisition unit 102, a first feature amount extraction unit 103, a second feature amount extraction unit 104, a comparison unit 105, an output unit 106, a sound shift information input reception unit 107, an audio A data changing unit 108 and an input receiving unit 109 are provided.
The first feature quantity extraction unit 103 includes first vibrato information acquisition means 1031, first incoming information acquisition means 1032, and first pitch change information acquisition means 1033.
The second feature amount extraction unit 104 includes second vibrato information acquisition means 1041, second incoming information acquisition means 1042, and second pitch change information acquisition means 1043.

The audio data storage unit 101 stores audio data to be imitated (hereinafter referred to as “teacher data” as appropriate). The audio data is, for example, MIDI format musical sound data, WAV format sound data, or the like. However, the format of the audio data is not limited. The voice data is singer's voice data, animal calls, machine sounds, voice data when reading English or Korean words and sentences, and the like. The audio data storage unit 101 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium.
The voice acquisition unit 102 acquires voice generated by a person and converts it into voice data. The sound acquisition unit 102 can be realized by, for example, a microphone and software that converts sound collected by the microphone into sound data.

  The first feature quantity extraction unit 103 extracts a predetermined feature quantity of the voice acquired by the voice acquisition unit 102. The predetermined feature amount is one or more feature amounts that a person feels to be similar if the degree of similarity is high. The predetermined feature amount includes, for example, one or more information among information on vibrato extracted from audio data, information on how to enter sound, and information on change in pitch. The first feature amount extraction unit 103 can usually be realized by an MPU, a memory, or the like. The processing procedure of the first feature quantity extraction unit 103 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The second feature amount extraction unit 104 extracts a predetermined feature amount from the audio data stored in the audio data storage unit 101. The feature quantity extracted by the second feature quantity extraction unit 104 is the same type of feature quantity as the feature quantity extracted by the first feature quantity extraction unit 103. The second feature amount extraction unit 104 can be usually realized by an MPU, a memory, or the like. The processing procedure of the second feature quantity extraction unit 104 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The comparison unit 105 compares the feature amount extracted by the first feature amount extraction unit 103 with the feature amount extracted by the second feature amount extraction unit 104 and passes the comparison result to the output unit 106. When comparing two or more feature amounts, the comparison unit 105 compares them for each feature amount. In such a case, the comparison result may be output for each feature amount, or one result may be generated and output based on two or more comparison results. The comparison result may be a voice imitation index indicating the degree of voice imitation of the entire voice imitation, or a comparison result for each part (for example, one measure). The comparison unit 105 can usually be realized by an MPU, a memory, or the like. The processing procedure of the comparison unit 105 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The output unit 106 outputs the result compared by the comparison unit 105. It is preferable that the output unit 106 outputs the result of comparison by the comparison unit 105 visually, preferably as an image. This is because the user can easily understand the comparison result. Further, it is preferable that the output unit 106 outputs the result of comparison performed by the comparison unit 105 in real time. This is because if the comparison is continued for a certain time or longer, the user can easily correct the image in a similar manner. Furthermore, it is more preferable that the output unit 106 changes the face image having the image of the eyes or / and the nose or / and the mouth, and displays it in a manner that the comparison result is good. The voice uttered by the user changes by changing the shape of the face (mainly changing the shape of the mouth). When a face for uttering in a similar manner is displayed, it becomes very easy to correct for the user to resemble the sound of the object to be recreated. Output usually refers to display on a display, but is a concept including printing on a printer, transmission to an external device, and the like. The output unit 106 may be considered as including or not including an output device such as a display. The output unit 106 can be realized by driver software of an output device or driver software of an output device and an output device.

  The sound misalignment information input accepting unit 107 accepts input of sound misalignment information, which is information indicating the degree to which the sound data stored in the sound data storage unit 101 is changed. The input means may be anything such as a numeric keypad, keyboard, mouse or menu screen. The sound shift information input accepting unit 107 can be realized by a device driver of an input means such as a numeric keypad or a keyboard, control software for a menu screen, or the like.

  The sound data changing unit 108 automatically changes the sound data stored in the sound data storage unit 101 based on the sound shift information received by the sound shift information input receiving unit 107. The algorithm for changing the voice data is not limited. When the sound shift information is ratio information, the audio data changing unit 108 changes, for example, the sound information of the ratio of sound shift information by a random value. The random value is obtained by, for example, a random number. The audio data changing unit 108 can be usually realized by an MPU, a memory, or the like. The processing procedure of the audio data changing unit 108 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

The input receiving unit 109 receives a process start instruction or an end instruction. When the start instruction is received, the sound acquisition unit 102 starts acquiring sound, and comparison between the stored sound data and the acquired sound data is started. Further, the processing of the information processing apparatus is ended by accepting the end instruction. The instruction input means may be anything such as a numeric keypad, a keyboard, a mouse, or a menu screen. The input receiving unit 109 can be realized by a device driver for input means such as a numeric keypad and a keyboard, control software for a menu screen, and the like.
The first vibrato information acquisition unit 1031 and the second vibrato information acquisition unit 1041 acquire vibrato information that is information on vibrato from the audio data. A specific example of vibrato information will be described later.
The first incoming information acquisition unit 1032 and the second incoming information acquisition unit 1042 acquire incoming information that is information relating to the way the sound enters from the voice data. A specific example of the input information will be described later.
The first pitch change information acquisition unit 1033 and the second pitch change information acquisition unit 1043 acquire pitch change information that is information related to pitch changes from the audio data. A specific example of the pitch change information will be described later.

Each of the above means can usually be realized by an MPU, a memory, or the like. The processing procedure of each means is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).
Hereinafter, an operation in which the information processing apparatus determines how much the voice generated by the user is similar to the voice data to be imitated and outputs the judgment result will be described with reference to the flowchart of FIG.
(Step S201) The input receiving unit 109 determines whether a start instruction has been received. If a start instruction is accepted, the process goes to step S202, and if not accepted, the process returns to step S201.
(Step S202) The voice acquisition unit 102 acquires voice generated by a person and converts it into voice data. The converted audio data is added to the buffer. The converted audio data is, for example, waveform data described later.

  (Step S203) The first feature quantity extraction unit 103 determines whether or not it is a segment for comparing audio data. If it is a delimiter, the process goes to step S204. If not, the process returns to step S202. Whether or not it is a delimiter is determined, for example, based on whether or not a predetermined time has elapsed. The predetermined time is preferably about 0.03 seconds as will be described later. In addition, the first feature amount extraction unit 103 may determine that the audio data acquired by the audio acquisition unit 102 is a delimiter when there is no audio for a predetermined time or longer.

(Step S204) The first vibrato information acquisition unit 1031 acquires first vibrato information from the audio data stored in the buffer. An example of an algorithm for acquiring vibrato information from audio data will be described later.
(Step S205) The first incoming information acquisition unit 1032 acquires first incoming information from the audio data stored in the buffer. An example of an algorithm for acquiring input information from audio data will be described later.
(Step S206) The first pitch change information acquisition unit 1033 acquires first pitch change information from the audio data stored in the buffer. An example of an algorithm for acquiring pitch change information from audio data will be described later.

(Step S207) The second vibrato information acquisition unit 1041 acquires the second vibrato information from the corresponding audio data in the audio data of the audio data storage unit 101. An example of an algorithm for acquiring vibrato information from audio data will be described later. Note that the corresponding audio data is audio data corresponding to the audio data acquired by the audio acquisition unit 102.
(Step S208) The second incoming information acquisition unit 1042 acquires the second incoming information from the corresponding voice data in the voice data storage unit 101. An example of an algorithm for acquiring input information from audio data will be described later.
(Step S209) The second pitch change information acquisition unit 1043 acquires second pitch change information from the audio data stored in the buffer. An example of an algorithm for acquiring pitch change information from audio data will be described later.
(Step S210) The comparison unit 105 compares the first vibrato information and the second vibrato information, and outputs a comparison result.
(Step S211) The comparison unit 105 compares the first input information and the second input information, and outputs a comparison result.
(Step S212) The comparison unit 105 compares the first pitch change information and the second pitch change information, and outputs a comparison result.
(Step S213) The comparison unit 105 calculates a score using the comparison result output in steps S210 to S212 as a parameter. The score calculated here is a partial voice imitation index.

(Step S214) The output unit 106 configures an image to be output based on the comparison results output in steps S210 to S212. The process of “composing an image” may be a process of reading stored image data.
(Step S215) The output unit 106 outputs the image configured in step S214.

  (Step S216) It is determined whether or not the process is finished. The determination of whether or not to end is performed by determining whether or not the input receiving unit 109 has received an end instruction or whether or not the audio data comparison process has ended. If completed, go to step S217, otherwise return to step S202. In addition, before returning to step S202, the pointer which reads the audio | voice data of the audio | voice data storage part 101 is shifted. That is, the pointer is moved to the head address of the next audio data to be compared.

(Step S217) The comparison unit 105 calculates an overall score from the one or more scores calculated in Step S213. The overall score is a voice imitation index. The calculation of the total score may be the sum of one or more scores calculated in step S213, may be an average, or may be corrected so that the total value becomes 100 points.
(Step S218) The output unit 106 outputs the voice mimic index calculated in step S217.

  Next, processing when the information processing apparatus receives input of sound shift information will be described. When the sound shift information input receiving unit 107 receives the input of the sound shift information, the sound data changing unit 108 automatically converts the sound data stored in the sound data storage unit 101 based on the sound shift information. Change to

  Next, an experiment performed at the preparation stage of development of the information processing apparatus will be described. In developing this information processing apparatus, it is important to extract feature quantities of voice imitation. Therefore, it is necessary to determine a standard for humans to feel similar to voice imitation. There are people who hear the same voice, but who do not look like it. This is because there are individual differences in the criteria for determining that the voice imitation is similar. However, there may be a common evaluation standard for certain characteristic sounds. Therefore, a voice imitation evaluation experiment was performed to extract the features of voice imitation.

  In this evaluation experiment, when the teacher imitates the teacher data as to whether there is a voice imitation that most evaluators judge to be similar, the voice imitation is the same sound as the teacher data. The purpose is to investigate whether it can be judged and how much the evaluation will be. Teacher data is data to be imitated and is voice data in the voice data storage unit 101 in the configuration of the information processing apparatus.

  As for the content of the experiment, first, short teacher data less than 3 seconds is listened only once. After that, they listened to 10 amateur voice imitations at 5 second intervals, and scored between 5 and 100 seconds with a score from 0 to 100 based on personal intuition. The evaluators were told that they were aiming to collect data for use in machine learning teacher data, without linking the true purpose of the experiment. For 5 types of sound data, 23 people conducted evaluation experiments. The evaluation results are shown in FIG.

  In the table of FIG. 3, the overall average is the average score of all evaluators. The voice imitation maximum is the average score of all graders of the highest score, the lowest is the average score of all graders of the lowest score, the TOP acquisition rate is the percentage of people who scored the highest, The teacher data recognition rate is a recognition rate and an average score when teacher data is mixed with a voice imitation list. The case where the highest score is assigned to the teacher data is regarded as “recognition”. Data that is not mixed with teacher data is indicated by “−”. The graders are all Japanese, and the people who imitate the voice are Malaysian, Swiss German, Italian, French and Japanese. In the table of FIG. 3, the first teacher data is an electronic sound of a dinosaur toy, and the second is a real goat cry. The third is a short Japanese sentence that says, "Oh my lord is bad." The fourth is a Swiss German word for cupboard. The fifth is long French.

  The person who imitated the electronic sound of the first dinosaur toy was one Malaysian man, two Swiss German men, one French man, two Japanese men, one Italian woman, There are a total of 10 Japanese women. The voice imitation data does not include teacher data. As a result of the evaluation experiment, only a Japanese woman got a high score. FIG. 4 shows the waveform data and time-frequency analysis results of the highly evaluated female data, the poorly evaluated data, and the teacher data. The top of FIG. 4 is a waveform. The frequency spectrum is shown in FIG. The bottom of FIG. 4 is a time-frequency analysis (x axis: time, y axis: frequency). A white portion is a portion having a large amplitude.

  From the analysis results, it can be seen from the analysis results that the voice imitation determined to be similar is similar to the teacher data and the frequency spectrum, the sound rise is similar, and the waveform is similar. However, the time is about 1 second longer than about 2 seconds of data. From this, regarding the electronic sound of the teacher's first dinosaur toy, it seems that the accuracy of the tempo is not included in the feature amount when judging whether or not they are similar.

  In the case of Goat of the teacher data 2 where the grader can recognize the feature amount of the teacher data, the recognition rate of the teacher data was as high as 82.6%. However, the Swiss data of teacher data 4 and 5 for which it was difficult to recognize the feature amount of the teacher data and French 39.1% and 30.4% were both low recognition rates. As for French in Teacher Data 5, the same voice imitation was mixed in the voice imitation list, but only four graders gave the same score. From this, it was found that although it is possible to score a sound whose feature value is easy to capture, it cannot be evaluated for a sound whose feature value cannot be captured. For foreign languages that have not been heard, it is difficult to capture the characteristics even with 3 seconds of data. For example, the reason why a sane person cannot capture the pitch is because the feature of the scale cannot be captured. I guess it is not. As for music, it is very difficult for people who are inexperienced in music and those who are not familiar with it to capture the pitch of a long passage at once. For this reason, it is considered that it is more effective to teach phonic correction teaching data at short time intervals by dividing each measure or melody.

  The voice imitation that everyone feels similar is a comparison of the feature value that each person remembers with the voice that is currently being heard, so the oversized person feels similar. From the evaluation experiment, it was found that humans cannot accurately memorize the sound itself, even if only a few seconds of data can not be captured. This is also inferred from the result that it cannot be judged when judging the voice imitation of a foreign language that has never been heard, or that all sounds the same, or teacher data cannot be identified as teacher data. For these reasons, even if you can imitate the voice just like the teacher data, there is a possibility that other people will not judge you to be similar. Therefore, it seems that it is more effective to use the processed data in which the feature amount of the teacher data is more remarkable as the data to be used for teaching.

  According to the experiments in FIG. 3 and FIG. 4, the feature amount was examined based on voice mimic data that was evaluated higher than the teacher data and voice mimic data that was generally highly evaluated. Specifically, we compared the features required for voice mimicry. As sound feature quantities, we extract the harmonic components, rise time, rise characteristics, vibrato, amplitude modulation, pitch fluctuation, etc. We examined whether such factors affect voice mimicry, and decided the actual features to be used. As a result, the feature values of the audio data to be judged to be similar were determined to be mainly three feature values of vibrato, sound input, and relative change in pitch. A person compares two pieces of voice data and determines that the two pieces of voice data are similar when such feature amounts are mainly similar. In addition, the time (tempo) shift does not affect the evaluation.

Moreover, in order to obtain the characteristics of vibrato, a time resolution of 0.03 seconds or more is required. If there is a time resolution of 0.03 seconds or more, it is possible to obtain characteristics of the onset of sound and the relative change in pitch. Therefore, the time resolution required for sound classification is 0.03 seconds here. However, the time resolution required for sound classification may be about 0.03 seconds. The feature amount includes the presence / absence of vibrato obtained from the amplitude change amount at each time, strength, presence / absence of crescendo, strength of attack (sound start), time difference in volume, and the like.
Hereinafter, a specific operation of the information processing apparatus according to the present embodiment will be described. First, in the audio data storage unit 101, raw waveform data of teacher data that is a target of voice imitation is stored.

  First, the second feature quantity extraction unit 104 obtains second vibrato information, second input information, and second pitch change information from the raw waveform data of the teacher data stored in the voice data storage unit 101. Specifically, the following processing is performed.

  That is, the second feature amount extraction unit 104 of the information processing apparatus reads the raw waveform data in the audio data storage unit 101. This raw waveform data is data as shown in FIG. Then, the second feature amount extraction unit 104 rectifies the read raw waveform data. Next, the second feature amount extraction unit 104 averages the rectified waveforms at intervals of 0.03 seconds. Next, the second feature amount extraction unit 104 extracts an actual utterance portion from the rise and fall of the sound (see FIG. 5B). Next, the second feature amount extraction unit 104 performs time-frequency analysis by short-time Fourier transform (STFT) at intervals of 0.03 seconds. And the 2nd feature-value extraction part 104 obtains the template of FIG.5 (c).

  Next, it is assumed that the user inputs a voice imitation start instruction. The information processing apparatus receives an input of a start instruction, and the voice acquisition unit 102 acquires voice data generated by the user. The acquired voice data is raw waveform data as shown in FIG.

Next, the first feature quantity extraction unit 103 of the information processing apparatus obtains a template (data as shown in FIG. 5C) by the same process as the second feature quantity extraction unit 104 as described above. When the first feature quantity extraction unit 103 obtains a template as shown in FIG. 5C, the entire length is matched with the template of teacher data (data in the voice data storage unit 101) (this process is referred to as “Normalize”). "). For example, if the length of the teacher data is 1 s and the voice imitation is only 0.8 s, the length is increased to 1 s, and conversely if it is as long as 1.2 s, the length is reduced to 1 s. In this state, the above template (voice imitation template—FIG. 5C) is created. Such a template has the same length as the teacher data template.
Through the above processing, a teacher data template (referred to as template 2) and a template obtained from user input speech (referred to as template 1) were obtained.

  Then, the first vibrato information acquisition unit 1031, the first input information acquisition unit 1032 and the first pitch change information acquisition unit 1033 respectively receive the first vibrato information and the first input information from the template 1 obtained from the user input voice. Information, first pitch change information.

Specifically, the first vibrato information acquisition unit 1031 sets the frequency (Fmax1) with the strongest amplitude of the template 1 for each unit time (the unit time is within one block of FIG. 5C and within 0.03 s). To obtain first vibrato information. The first vibrato information is a numeric string of frequencies (Fmax1). The second vibrato information acquisition unit 1041 calculates the frequency (Fmax2) with the strongest amplitude of the template 2 for each unit time, and obtains the second vibrato information. The second vibrato information is also a numeric string of frequencies (Fmax2).

  Next, the first input information acquisition unit 1032 acquires the amplitude at each frequency of the first predetermined number (for example, 5) blocks of the template 1. Further, the second input information acquisition unit 1042 acquires the amplitude at each frequency of the first predetermined number (for example, 5) blocks of the template 2.

Next, the first pitch change information acquisition unit 1033 acquires the frequency of the strongest amplitude for each unit time of the template 1. That is, the first pitch change information is an amplitude frequency sequence. The second pitch change information acquisition unit 1043 acquires the frequency of the strongest amplitude for each unit time of the template 2. That is, the second pitch change information is also a frequency train of amplitude.
The comparison unit 105 compares the vibrato information, the incoming information, and the pitch change information of the template 1 and the template 2 described above.

  First, the comparison unit 105 compares the first vibrato information and the second vibrato information, and calculates the similarity between the teacher data and the voice vibrato input by the user. The vibrato similarity is an example of information on the vibrato described above. Specifically, the comparison unit 105 calculates the vibrato similarity using the difference in position and the number of data of two templates as parameters. The comparison unit 105 may calculate the similarity by machine learning using an artificial neural network (ANN). That is, the comparison unit 105 determines based on the questionnaire data. For example, the load is determined by learning ANN from teacher data such as A's voice imitation score average of 60, B's voice imitation score average of 70, etc., and then the unknown voice imitation data C's score is obtained. .

  Hereinafter, machine learning by ANN will be described. An input of ANN is a feature amount (information obtained from a template), and an output is a score. If A's voice imitation feature is input and A's average score is 60 points, the ANN load is learned until the ANN output reaches 60 points. Data used for such learning is called a pattern signal. Since there is a bias in learning with a single pattern signal, the same learning is performed on Mr. B and Mr. C using the same neural network, and further learning of the load is performed (for example, five patterns). Thus, the learned ANN is prepared in advance on the system side, and the score of the user's voice imitation is output by the ANN. In other words, the ANN corresponds to, for example, several average judges.

  Further, the comparison unit 105 compares the first input information with the second input information, and calculates the similarity of the sound input between the teacher data and the voice input by the user. In addition, the similarity of the way of entering a sound is an example of information related to the way of entering a sound. Specifically, it is assumed that the amplitude at each frequency of the first five blocks of the template 1 is the first input information. Further, it is assumed that the amplitude at each frequency of the first five blocks of the template 2 is the second input information. Then, the comparison unit 105 assumes that the number obtained by multiplying the reciprocal of the difference between each element of the first input information and each element of the second input information by a predetermined integer is the similarity between the two pieces of input information. The degree of similarity regarding how to enter the sound is calculated.

Further, the comparison unit 105 compares the first pitch change information and the second pitch change information, and calculates an overall similarity that is an overall similarity tendency between the teacher data and the voice input by the user. . This similarity is an example of information related to a change in pitch. Specifically, the comparison unit 105 multiplies a reciprocal of the sum of the difference between the frequency sequence of the amplitude that is the first pitch change information and the frequency sequence of the amplitude that is the second pitch change information by a predetermined integer. The number is calculated as the degree of similarity related to the change in pitch.
Note that the frequency having the strongest amplitude is calculated by, for example, an average of the frequencies.

  Further, the comparison unit 105 calculates a voice imitation index based on the above-described vibrato similarity, sound entry similarity, and similarity regarding pitch change. Specifically, for example, the comparison unit 105 calculates the total of three similarities as a voice imitation index. The comparison unit 105 may calculate an average value of the three similarities as a voice imitation index.

  Next, the output unit 106 holds, for example, an output face image determination table shown in FIG. 6 and one or more output face images shown in FIG. The output face image determination table holds one or more records having “ID”, “condition”, and “image ID”. “ID” is information for identifying a record and exists for a request in table management. “Condition” is a condition for determining an image to be output using a feature amount as a parameter. If the attribute value of “condition” matches the result calculated by the comparison unit 105, the image of “image ID” is output. “Image ID” is an identifier for identifying an image. The output face images in FIG. 7 are four images here. The image of “ID = 1” in the four images is an image that is displayed when teaching the softening of sound. The image of “ID = 2” is an image that is displayed when teaching to make the sound hard. The image with “ID = 3” is an image displayed when teaching to weaken the vibrato. The image of “ID = 4” is an image displayed when teaching to increase vibrato.

  The output unit 106 selects and displays an image according to the result of comparison by the comparison unit 105 in light of the conditions of the output face image determination table. If the comparison result by the comparison unit 105 is ““ first vibrato information ”−“ second vibrato information ”= 12”, it matches the condition of the record of “ID = 3” in the output face image determination table. Then, the image of “Image ID = 3” is selected and displayed. Such information selection and display are continuously performed in real time by the information processing apparatus while the user inputs voice.

Further, the output unit 106 outputs the voice imitation index calculated above in a manner as shown in FIG. When the user clicks the “Midi Open” button 1 on the screen of FIG. 8, the information processing apparatus reads the data for MIDI reproduction. When the user presses the “PLAY” button 2, the information processing apparatus displays a spectral display screen (large black square in the center of FIG. 8) 3 with a shaded square and a concentric spectrum display screen (round black on the right side of FIG. 8). The correct pitch is taught with a black circle on the screen. When the user sings, each display screen is similarly displayed in a first predetermined color (for example, orange). If the sung pitch is correct, the second predetermined color (for example, yellow) changes the teaching color. When the pitch is shifted, the face image (lower right face 4 in FIG. 8) teaches that the face image teaches “higher” and “lower” (see FIG. 12). When the pitch is correct, an expressionless face image is displayed as shown in FIG. With the above display, the user can practice voice imitation while correcting the trajectory in real time.
Next, the operation of the information processing apparatus for acquiring banquet art for singing a song forcibly will be described.

  Now, it is assumed that voice data of a song sung by a singer is stored in the voice data storage unit 101 of the information processing apparatus. In such a situation, the user inputs sound shift information that is information indicating the degree to which the sound data is changed. Here, the sound misalignment information includes sound misalignment information which is information indicating the frequency of occurrence of sound misalignment and a sound misalignment level indicating the width (size) of the sound misalignment. When the user sets the sound shift information to “50%” and the sound shift level to “7” (see the left side of FIG. 8), the sound shift information input receiving unit 107 receives the sound shift information. Next, the audio data changing unit 108 automatically changes the audio data stored in the audio data storage unit 101 based on the sound deviation information “50%” and the sound deviation level “7”. That is, the audio data changing unit 108 changes the audio data so that 50% of the audio data in the audio data storage unit 101 is increased or decreased by a maximum of “7” intervals.

Specifically, for example, as shown in FIG. 9A, it is assumed that the original teacher data is 12 scales. Then, under the condition that 50% of all data has a sound shift, the sound data changing unit 108 determines data to be shifted as shown in FIG. 9B. There is no limitation on the algorithm used by the voice data changing unit 108 to determine the data to be shifted. The audio data changing unit 108 may determine data to be shifted by one skip, or may generate random numbers and determine data to be shifted by the generated random numbers. In FIG. 9B, the data that is shifted in sound is underlined data. Next, the audio data changing unit 108 determines the degree of sound deviation so that the original sound is shifted by ± 7 at the maximum. There is no limitation on the algorithm by which the audio data changing unit 108 determines the degree of sound deviation. For example, the audio data changing unit 108 generates a random number and assigns numbers from “−7” to “+7” by the remainder obtained by dividing the generated random number by 14. Then, the voice data changing unit 108 obtains changed teacher data as shown in FIG. Then, the user learns banquet performance to sing a song compulsorily by practicing voice imitation based on such changed teacher data (data of a song that is a sound of the original beautiful song) be able to. The operation of the information processing apparatus in voice imitation is as described above.
As described above, according to the present embodiment, voice imitation can be practiced easily. In addition, the ability to imitate voices that people feel similar to can be obtained.
Further, according to the present embodiment, the degree of similarity can be displayed in real time with an index that is close to the index that a person feels, and for example, when a song is imitated, the trajectory can be corrected in the middle. .

  Furthermore, according to the present embodiment, for example, correct voice data of a song can be forcibly changed to sound data, and the similarity to the sound data can be output, and the song can be forcibly sung. You can learn banquet arts.

  In this embodiment, the voice imitation includes singing imitation, animal noise imitation, mechanical sound imitation, language pronunciation imitation, and the like. That is, the teacher data in the present embodiment is singer singing voice data, animal cry data, machine sound data, language pronunciation data, and the like. The same applies to other embodiments.

  Further, according to the present embodiment, the feature quantity for determining voice imitation is effective as vibrato information, incoming information that is information on how to enter sound, and pitch change information that is information related to pitch change. However, other feature amounts may be used for calculating the voice imitation index. This also applies to other embodiments.

  Further, according to the present embodiment, an operation for obtaining a template of teacher data is performed in real time while obtaining voice from the user and comparing the two templates. However, the operation of obtaining the teacher data template may be performed in advance before obtaining the voice from the user. An operation for obtaining a template of teacher data may be performed in advance by another device, and the information processing device may hold the feature amount of the teacher data extracted by the other device. In this case, the information processing apparatus includes a sound acquisition unit that acquires sound, a first feature amount extraction unit that extracts a predetermined feature amount of the sound acquired by the sound acquisition unit, and a second feature that extracts a predetermined feature amount. A feature amount extraction unit; a comparison unit that compares the feature amount extracted by the first feature amount extraction unit; and a feature amount of audio data for comparison; and an output unit that outputs a result of the comparison by the comparison unit. Device.

Further, according to the present embodiment, the information processing apparatus operates stand-alone , but may operate in a server / client system. This also applies to other embodiments. In this case, the information processing system has the system configuration shown in FIG. That is, the information processing system includes a client device 91 and a server device 92. The client device 91 includes a voice acquisition unit 102, an output unit 106, a sound shift information input reception unit 107, an input reception unit 109, and a first transmission / reception unit 9101. The server device 92 includes a second transmission / reception unit 9201, an audio data storage unit 101, a first feature quantity extraction unit 103, a second feature quantity extraction unit 104, a comparison unit 105, and an audio data change unit 108. The first transmission / reception unit 9101 of the client device 91 transmits the voice data generated by the user to the server device 92. The second transmission / reception unit 9201 of the server device 92 receives the audio data. The comparison unit 105 compares one or more feature amounts between the received audio data and the stored audio data. The second transmission / reception unit 9201 transmits the comparison result to the client device 91. Next, the first transmission / reception unit 9101 of the client device 91 receives the comparison result, and the output unit 106 outputs it. That is, this is a mode in which the processing of the information processing device described above is distributed and processed by the client device 91 and the server device 92.

  Further, according to the present embodiment, when the information processing apparatus calculates the voice imitation index, the voice data stored in the voice data storage unit 101 is not output as a voice. good. It is often preferable to output the voice data as voice, which helps imitate the user's voice.

  Further, according to the specific example in the present embodiment, the output unit 106 changes the face image having the image of the eyes or / and the nose or / and the mouth so that the result of voice mimicry is improved. Although the image is displayed, the comparison result of voice imitation may be displayed by displaying an image other than the face image. That is, it is only necessary to visualize the difference (difference data) in the feature amount between the voice uttered by the user and the sound to be compared (voice data in the voice data storage unit 101). For the display of the difference data, instead of the “face” as shown in FIG. 7, “thorn” shown in FIG. 11A, “cone” shown in FIG. 11B, “shown in FIG. 11C” "Ball" etc. may be used. A “thorn” is an object in which twelve thorns extend from three spheres, and the length of each thorn represents the size of the difference data. “Cone” has twelve cones rotating in a circle, and the length of each cone represents the size of the difference data. “Ball” has 8 spheres that rotate counterclockwise on the outer periphery and 4 spheres that rotate clockwise on the inner periphery, respectively. expressing. Note that the “face” shown in FIG. 7 expresses the size of the difference data by the difference between the size and position of each part of the face expressed by eyes, nose, and mouth with a prescribed size and location. Further, there may be variations in which the facial expression changes greatly when the difference data exceeds a certain value for the face. For example, when the pitch is very low compared to the teacher data, a “face” as shown in FIG. 12A is displayed, and the “pitch is made higher” is intuitively taught, or the pitch is the teacher. If it is very high compared to the data, a “face” as shown in FIG. 12B may be displayed, and “to make the sound lower” may be intuitively taught. Differences in feature quantities are displayed in correspondence with intuitive intelligibility and real-time input, which is suitable for improving voice imitation.

  Furthermore, the processing in the present embodiment may be realized by software. Then, this software may be distributed by software download or the like. Further, this software may be recorded and distributed on a recording medium such as a CD-ROM. This also applies to other embodiments in this specification. Note that the software that implements the information processing apparatus according to the present embodiment is the following program. That is, the program includes: a sound acquisition step for acquiring sound; a first feature amount extraction step for extracting a predetermined feature amount of the sound acquired in the sound acquisition step; and the first feature amount extraction unit. This is a program for executing a comparison step for comparing the extracted feature amount with the feature amount of the audio data for comparison and an output step for outputting the result of comparison in the comparison step.

In addition, the program stores, in a computer, a sound acquisition step for acquiring sound, a first feature amount extraction step for extracting a predetermined feature amount of the sound acquired in the sound acquisition step, and a predetermined amount from stored sound data. A second feature amount extraction step for extracting the feature amount, a comparison step for comparing the feature amount extracted in the first feature amount extraction step, the feature amount extracted in the second feature amount extraction step, and the comparison step This is a program for executing an output step for outputting the result of comparison in (1).
(Embodiment 2)
In the present embodiment, an information processing apparatus that can practice voice imitation on stored voice data, and that can display voice imitation index of each part of the voice data, and can perform voice imitation on a part Device.

  FIG. 13 is a block diagram of the information processing apparatus according to this embodiment. The information processing apparatus includes an audio data storage unit 101, an audio acquisition unit 102, a first feature quantity extraction unit 103, a second feature quantity extraction unit 104, a comparison unit 1105, an output unit 1106, a sound shift information input reception unit 107, an audio A data changing unit 108, an input receiving unit 1109, and an audio output unit 1110 are provided.

  The comparison unit 1105 compares the feature amount extracted by the first feature amount extraction unit 103 with the feature amount extracted by the second feature amount extraction unit 104 for each part of the audio data. The voice data is, for example, data of a song sung by a singer. The audio data portion is cut out, for example, when a predetermined output time elapses. The predetermined time is preferably about 0.03 seconds as described above. In addition, the audio data may be determined to be a delimiter when there is no sound for a certain time or longer. Note that the comparison unit 1105 may compare the two or more feature amounts extracted by the first feature amount extraction unit 103 with the two or more feature amounts extracted by the second feature amount extraction unit 104 for each feature amount. The comparison unit 1105 can be usually realized by an MPU, a memory, or the like. The processing procedure of the comparison unit 1105 is usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The output unit 1106 outputs a comparison result for each part output by the comparison unit 1105. The comparison result may be indicated by a score or an image. Output usually refers to display on a display, but is a concept including printing on a printer, transmission to an external device, and the like. The output unit 1106 may or may not include an output device such as a display. The output unit 1106 can be realized by output device driver software, or output device driver software and an output device.

  The input receiving unit 1109 receives an input for instructing a process start instruction, an end instruction, or a part. The “input for instructing a part” is, for example, an instruction input for the comparison result for each part output by the output unit 1106, and is an instruction input for the part. A portion is a portion of audio data. The instruction input means may be anything such as a numeric keypad, a keyboard, a mouse, or a menu screen. The input receiving unit 1109 can be realized by a device driver for input means such as a numeric keypad or a keyboard, control software for a menu screen, and the like.

  The audio output unit 1110 reads out a part of audio data corresponding to the instructed part and outputs the audio. Such audio data is usually audio data in the audio data storage unit 101. However, the voice data may be voice data that is input by the user first and acquired by the voice acquisition unit 102.

When the input receiving unit 1109 receives an input indicating a part, the voice acquisition unit 102 acquires the voice, and the first feature amount extraction unit 103 is a predetermined feature amount of the voice acquired by the voice acquisition unit 102. The second feature amount extraction unit 104 extracts a predetermined feature amount from a part of the audio data, and the comparison unit 1105 extracts one or more feature amounts extracted by the first feature amount extraction unit 103 and the second feature amount. One or more feature amounts extracted by the feature amount extraction unit 104 are compared, and an output unit 1106 outputs a result of comparison by the comparison unit 1105.
Hereinafter, the operation of the information processing apparatus for outputting the voice imitation index of each part will be described with reference to the flowchart of FIG. In the flowchart of FIG. 14, the description of the same processing as that of the flowchart of FIG. 2 is omitted.

(Step S1401) The output unit 106 outputs the score calculated in step S213. This score is a comparison result for each part. The display mode and display timing of the comparison result for each part are not limited. The display mode of the score for each part may be based on the above-described face image, or the score of the voice imitation for each part may be displayed as a numerical value.
Next, the operation when the information processing apparatus is used for practicing partial voice imitation will be described with reference to the flowchart of FIG.

(Step S1501) The input receiving unit 1109 determines whether or not an input indicating a part has been received. If an input indicating a part is accepted, the process proceeds to step S1502, and if an input indicating a part is not accepted, the process returns to step S1501.
(Step S1502) The audio output unit 1110 reads audio data corresponding to the portion indicated by the input received in step S1501 from the audio data storage unit 101.
(Step S1503) The audio output unit 1110 outputs the audio data read in step S1502.

(Step S1504) The voice acquisition unit 102 acquires voice generated by the user. Note that the audio data output in step S1503 and the audio acquisition in step S1504 are preferably executed in parallel. In addition, when this loop is repeated twice or more, the acquired sound is appended.
(Step S1505) It is determined whether or not all the outputs of the portion indicated by the input received in step S1501 have been completed. If completed, go to step S1506, and if not completed, return to step S1502.

  (Step S1506) The first feature quantity extraction unit 103 extracts the first feature quantity from the voice acquired in step S1504. The first feature amount is, for example, the vibrato information, the incoming information, and the pitch change information described in the first embodiment, but may be another feature amount.

(Step S1507) The second feature quantity extraction unit 104 extracts a second feature quantity from the audio data read in step S1502. The second feature amount is, for example, the vibrato information, the input information, and the pitch change information described in the first embodiment, but may be another feature amount.
(Step S1508) The comparison unit 1105 compares the first feature value acquired in Step S1506 with the second feature value acquired in Step S1507.
(Step S1509) The output unit 1106 outputs the comparison result in step S1508. The process ends.
Through the above processing, the user can practice, for example, imitation of the sound data of a song stored in the audio data storage unit 101 for some phrases (for example, one measure).

  Hereinafter, a specific operation of the information processing apparatus according to the present embodiment will be described. First, in the audio data storage unit 101, raw waveform data of teacher data that is a target of voice imitation is stored. Here, the teacher data is voice data of a song.

  Then, the user inputs an instruction to start singing. Next, the information processing apparatus receives an input of a start instruction, and the voice acquisition unit 102 acquires voice data generated by the user. The acquired voice data is raw waveform data as shown in FIG.

Then, the first feature amount extraction unit 103 of the information processing apparatus performs the same process as the process described in Embodiment 1 on the voice acquired by the voice acquisition unit 102. And the 1st feature-value extraction part 103 acquires 1st vibrato information, 1st incoming information, and 1st pitch change information.
Next, the second feature amount extraction unit 104 obtains second vibrato information, second input information, and second pitch change information from the raw waveform data of the teacher data stored in the voice data storage unit 101. .

  Then, the comparison unit 105 compares the first vibrato information and the second vibrato information, and calculates the similarity between the teacher data and the voice vibrato input by the user. Further, the comparison unit 105 compares the first input information with the second input information, and calculates the similarity of the sound input between the teacher data and the voice input by the user. Further, the comparison unit 105 compares the first pitch change information and the second pitch change information, and calculates an overall similarity that is an overall similarity tendency between the teacher data and the voice input by the user. . Further, the comparison unit 105 calculates a voice imitation index based on the above-described vibrato similarity, sound entry similarity, and overall similarity. The voice imitation index is a score of 100 points. Then, the output unit 106 outputs a voice imitation index. Since this process is the same as the process described in the first embodiment, detailed description thereof is omitted.

Such a process is repeated for each measure. The result is shown in FIG. In FIG. 16, while the user sings a song, a voice imitation index, which is the degree of imitation of the song, is output one measure at a time in real time. Further, the output unit 106 indicates the bars lower than the predetermined score by shading.
Next, it is assumed that the user designates “PhraseNo.” With shaded points on the display of FIG. This instruction is the above-described “input for instructing a part”.

Next, the input receiving unit 1109 receives an input for instructing the portion. Then, the audio output unit 1110 reads out the audio data corresponding to the portion “PhraseNo. = 2” indicated by the received input from the audio data storage unit 101. Next, the audio output unit 1110 outputs audio data of “PhraseNo. = 2”. In the meantime, the user speaks in order to practice singing a measure corresponding to “PhraseNo. = 2”. Meanwhile, the voice acquisition unit 102 acquires voice generated by the user.
Next, the first feature amount extraction unit 103 utters the user and extracts the first feature amount from the acquired voice. The feature amounts are vibrato information, incoming information, and pitch change information.

Next, the second feature amount extraction unit 104 extracts a second feature amount from the read audio data. The second feature amount is also vibrato information, incoming information, and pitch change information. Next, the comparison unit 1105 compares the first feature value and the second feature value. Then, the output unit 1106 outputs the comparison result (see FIG. 17).
As described above, according to the present embodiment, voice imitation can be practiced easily. In particular, according to the present embodiment, it is easy to practice a part of voice imitation. As a result, the ability to imitate voices that people feel similar to can be obtained.

  In the present embodiment, the face image is not displayed as in the first embodiment, but the face image and other images are displayed in the same manner as in the first embodiment while the voice is imitated. Thus, the imitation index may be presented to the user in an easy-to-understand manner.

  Further, according to the specific example in the present embodiment, for example, the function of forcibly changing the voice data of the correct song to the voice data forcibly and forcibly singing the song forcibly was not described. However, similar to the function described in Embodiment Mode 1, such a function may be provided. Such a function can be performed by the sound shift information input receiving unit 107 and the audio data changing unit 108.

  Furthermore, the software that implements the information processing apparatus according to the present embodiment is the following program. That is, the program includes: a sound acquisition step for acquiring sound; a first feature amount extraction step for extracting a predetermined feature amount of the sound acquired in the sound acquisition step; and the first feature amount extraction unit. A program for executing a comparison step for comparing the extracted feature amount and the feature amount of the comparison audio data for each portion of the audio data, and an output step for outputting a comparison result for each portion compared in the comparison step It is.

In the above program, the sound acquisition step or the like does not include processing performed by hardware, for example, processing performed by a speaker or the like in the sound acquisition step (processing performed only by hardware).
Moreover, the computer which performs said program may be single, and plural may be sufficient as it. That is, centralized processing may be performed, or distributed processing may be performed.

  In each of the above embodiments, the voice imitation index may be calculated by the following algorithm. That is, the details of the algorithm when the comparison unit of the information processing apparatus calculates the voice imitation index based on the similarity of vibrato, the similarity of how to enter sound, and the similarity related to the change in pitch will be described below.

  First, the second feature amount extraction unit of the information processing apparatus performs the following preprocessing. Now, it is assumed that the teacher data (a) is stored in the voice data storage unit. That is, (a) is a raw waveform (see FIG. 18). First, the second feature quantity extraction unit sets a value equal to or smaller than a certain value (here, a threshold value 0.05) to zero, reduces noise, and obtains FIG. 18B. Next, the second feature amount extraction unit rectifies the waveform with reduced noise to obtain FIG. Next, the second feature amount extraction unit takes an addition average with a window width. However, the length should be 0.03 seconds or less so that vibrato can be taken. As a result, the second feature quantity extraction unit obtains the data of FIG. And the 2nd feature-value extraction part cuts a part without a sound, in order to make a template. Then, even if the second feature quantity extraction unit is interrupted in the middle, the rear part is cut to obtain FIG. As described above, the second feature amount extraction unit extracts the sound start t1 and the end t2.

Next, the second feature quantity extraction unit performs STFT (Short Time Fourier Transform) with a window width of 0.01 within the range of t1 and t2 extracted from the data of FIG. 18E, and FIG. obtain. Next, the second feature quantity extraction unit extracts only the frequency having the maximum value at each time from the STFT result of (f), and obtains FIG. 19 (f). Further, the second feature amount extraction unit obtains the frequency having the largest maximum value from (g), extracts 10 components in descending order only within one octave above and below that frequency, and FIG. obtain.
Next, the first feature quantity extraction unit extracts 10 components within one upper and lower octave from the frequency having the maximum value of the voice mimic data by an algorithm similar to the algorithm of the second feature quantity extraction unit described above, and FIG. i) is obtained.

  Next, the comparison unit compares the two templates (FIG. 19 (h) and FIG. 19 (i)) as follows. First, the comparison unit compares the degree of similarity of how the sound enters. That is, the comparison unit compares data for only the first 10 blocks in FIG. 19H (for 0.1 seconds). A square portion surrounded by T_B in FIG. 19H corresponds to this. The same time T_B is extracted from FIG. 19 (i), and the difference for each time is taken. In the case of this data, it is assumed that the total error average value “diff = 0.0197” is obtained. When the comparison unit determines that the sound is similar, “Threshold 1 (−x) <diff <Threshold 2 (+ x)”, and when the comparison unit determines that the sound is weak, “Threshold 1 > diff ”, and when the comparison unit determines that the sound is strong, it is assumed that“ Threshold2 <diff ”. The data given in the above example is judged to be very similar. In addition, it is the 1st input information acquisition means and the 2nd input information acquisition means that acquire the information regarding how to enter the sound.

Next, the comparison unit compares the similarity between the first vibrato information and the second vibrato information. That is, the comparison unit checks whether or not it exists along the time axis at each frequency from the template in FIG. 19H (arrow in FIG. 19H). The comparison unit checks the on-off repetition time width of a certain frequency (for several blocks around the strongest frequency). When there is no on-off repetition, the comparison unit determines that “no vibrato” . Then, when the on-off repeats, the comparison unit determines that “vibrato exists”. Then, in the case of “with vibrato”, the comparison unit obtains vibrato Δt ((h) Δt).
Next, the comparison unit obtains vibrato Δt ′ based on the voice mimic data as described above.

Then, in the case of “Δt ′ >> Δt”, the comparison unit determines that the vibrato should be made finer, and the output unit outputs an instruction to make the vibrato finer. The comparison unit determines that the vibrato should be increased when “Δt ′ << Δt” or does not exist, and the output unit outputs an instruction to increase the vibrato. Further, when “Δt ′ == Δt”, the comparison unit determines that the vibrato information is similar, and the output unit outputs nothing or is good.
At the time of output, the similarity of sound input and the similarity of vibrato may be indexed and weighted to output one numerical value. Such one numerical value is, for example, a voice imitation index.

Further, the voice imitation index may be calculated as follows. That is, the comparison unit obtains a difference template from FIG. 18 (h) and FIG. 18 (i) (FIG. 21 (l)). From this difference template, the average difference data at each time is used as the input of the ANN. Then, three patterns with good results, normal ones and bad ones obtained from the questionnaire results are used as teaching data for ANN learning. There are 72 ANN input data, for example. The output shall be normalized by dividing the average score obtained from the questionnaire result by 100 points. The ANN learning is an error back propagation method. When the average difference data (m) obtained at this time is input to this learned ANN, the ANN outputs an imitation index. FIG. 22 is a model diagram of the ANN. In this case, it was 80 points.
It should be noted that the determination of the similarity between the first vibrato information and the second vibrato information may also be performed for STFT twice.

  The algorithm described above is an algorithm for calculating a voice imitation index based on two similarities among the similarity of vibrato, the similarity of how to enter a sound, and the similarity related to a change in pitch. It is an example of an algorithm for calculating a mimic index, and it goes without saying that other algorithms may be used.

In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.
The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the information processing apparatus according to the present invention has an effect of being able to practice voice imitation, and is useful as, for example, a device installed in a karaoke apparatus.

Block diagram of information processing apparatus according to Embodiment 1 Flow chart for explaining the operation of the information processing apparatus The figure which shows the table of the evaluation result of the same voice imitation The figure which shows the evaluation result of the same voice imitation Illustration explaining the algorithm for calculating the imitation index The figure which shows the output face image judgment table Figure showing the same output face image The figure which shows the example of output screens such as the same voice imitation index The figure explaining the change of the audio data Block diagram of the system configuration of the information processing system Figure showing an image showing the imitation index Figure showing the same output face image FIG. 10 is a block diagram of an information processing device in a second embodiment Flow chart for explaining the operation of the information processing apparatus Flow chart for explaining the operation of the information processing apparatus The figure which shows the example of output screens such as the same voice imitation index The figure which shows the display screen example of the same comparison result The figure explaining the specific example of the data conversion A specific example of the data conversion will be described. A specific example of the data conversion will be described. A specific example of the data conversion will be described. Model diagram explaining the ANN

Explanation of symbols

91 Client device 92 Server device 101 Audio data storage unit 102 Audio acquisition unit 103 First feature value extraction unit 104 Second feature value extraction unit 105, 1105 Comparison unit 106, 1106 Output unit 107 Sound shift information input reception unit 108 Audio data change Unit 109, 1109 input reception unit 1031 first vibrato information acquisition unit 1032 first input information acquisition unit 1033 first pitch change information acquisition unit 1041 second vibrato information acquisition unit 1042 second input information acquisition unit 1043 second pitch change information acquisition Means 1110 Audio output unit 9101 First transmission / reception unit 9201 Second transmission / reception unit

Claims (13)

An information processing device that performs voice imitation evaluation,
An audio acquisition unit for acquiring audio;
A first feature quantity extraction unit for extracting a predetermined feature quantity of the voice acquired by the voice acquisition unit;
A feature quantity the first feature extraction unit has extracted, a comparison unit for comparing the feature amount of the audio data to be compared,
An information processing apparatus including an output unit that outputs a result of comparison by the comparison unit;
The feature quantity that the comparison unit compares has information on how to enter the sound ,
The first feature amount extraction unit includes:
From the sound acquired by the sound acquisition unit, comprising first input information acquisition means for acquiring first input information that is amplitude at each frequency of the first predetermined number of blocks,
The comparison unit includes:
The second input information that is the amplitude at each frequency of the first predetermined number of blocks of the audio data to be compared is compared with the first input information, and the similarity of the sound input is obtained, An information processing apparatus that obtains the result of the comparison using the similarity of how sound enters. The feature amount compared by the comparison unit further includes information regarding a change in pitch,
The first feature amount extraction unit includes:
From the voice acquired by the voice acquisition unit further comprises first pitch change information acquisition means for acquiring first pitch change information that is a set of frequencies having the strongest amplitude per unit time,
The comparison unit includes:
Compare the second pitch change information and the first pitch change information, which is a set of frequencies of the strongest amplitude for each unit time of the audio data to be compared, and obtain the similarity regarding the pitch change, The information processing apparatus according to claim 1, wherein the comparison result is acquired using a similarity degree related to the change in pitch. The feature amount compared by the comparison unit further includes information on vibrato,
The first feature amount extraction unit includes:
From the voice acquired by the voice acquisition unit, further comprising a first vibrato information acquisition means for acquiring first vibrato information that is a set of frequencies having the strongest amplitude per unit time,
The comparison unit includes:
It is a set of frequencies acquired from the audio data to be compared, and the second vibrato information, which is a set of frequencies having the strongest amplitude per unit time, is compared with the first vibrato information, and similarities related to vibrato The information processing apparatus according to claim 1, wherein the information processing apparatus acquires a degree and obtains the result of the comparison using a similarity degree related to the vibrato. An information processing device that performs voice imitation evaluation,
An audio data storage unit storing audio data;
An audio acquisition unit for acquiring audio;
A first feature quantity extraction unit for extracting a predetermined feature quantity of the voice acquired by the voice acquisition unit;
A second feature quantity extraction unit for extracting a predetermined feature quantity from the voice data stored in the voice data storage unit;
A comparison unit that compares the feature amount extracted by the first feature amount extraction unit with the feature amount extracted by the second feature amount extraction unit;
An information processing apparatus including an output unit that outputs a result of comparison by the comparison unit;
Feature quantity the comparison unit compares the possess information regarding how to enter sound,
The first feature amount extraction unit includes:
From the sound acquired by the sound acquisition unit, comprising first input information acquisition means for acquiring first input information that is amplitude at each frequency of the first predetermined number of blocks,
The second feature amount extraction unit includes:
From the audio data stored in the audio data storage unit, comprising second input information acquisition means for acquiring second input information that is the amplitude at each frequency of the first predetermined number of blocks,
The comparison unit includes:
An information processing apparatus that compares the second incoming information with the first incoming information, obtains the similarity of how to enter the sound, and obtains the result of the comparison using the similarity of the entered sound . The feature amount compared by the comparison unit further includes information regarding a change in pitch,
The first feature amount extraction unit includes:
From the voice acquired by the voice acquisition unit further comprises first pitch change information acquisition means for acquiring first pitch change information that is a set of frequencies having the strongest amplitude per unit time,
The second feature amount extraction unit includes:
Further comprising second pitch change information acquisition means for acquiring second pitch change information that is a set of frequencies having the strongest amplitude per unit time from the voice data stored in the voice data storage unit;
The comparison unit includes:
The second pitch change information and the first pitch change information are compared, a similarity related to a pitch change is acquired, and the comparison result is also acquired using the similarity related to the pitch change. 4. The information processing apparatus according to 4. The feature amount compared by the comparison unit further includes information on vibrato,
The first feature amount extraction unit includes:
From the voice acquired by the voice acquisition unit, the frequency with the strongest amplitude is calculated per unit time, further comprising first vibrato information acquisition means for acquiring first vibrato information that is a set of the frequencies,
The second feature amount extraction unit includes:
A second vibrato information acquisition means for calculating a frequency with the strongest amplitude per unit time from the audio data stored in the audio data storage unit and acquiring second vibrato information that is a set of the frequencies; Equipped,
The comparison unit includes:
The said 2nd vibrato information and said 1st vibrato information are compared, the similarity regarding vibrato is acquired, and the said comparison result is acquired also using the similarity regarding the said vibrato. Information processing device. The information processing apparatus according to claim 1, wherein the feature quantity compared by the comparison unit does not include information on a tempo. The unit time is
The information processing apparatus according to claim 1, wherein the information processing apparatus is approximately 0.03 seconds. A program for assessing voice imitation,
On the computer,
An audio acquisition step for acquiring audio;
A first feature amount extraction step for extracting a predetermined feature amount of the voice acquired in the voice acquisition step;
A feature amount extracted by the first feature extraction step, a comparison step of comparing the feature amount of the audio data to be compared,
A program for executing an output step of outputting a result of comparison in the comparison step;
The feature amount to be compared in the comparison step has information on how to enter the sound,
The first feature amount extraction step includes:
From the sound acquired in the sound acquisition step, comprising a first input information acquisition step of acquiring first input information that is the amplitude at each frequency of the first predetermined number of blocks,
The comparison step includes
The second input information that is the amplitude at each frequency of the first predetermined number of blocks of the audio data to be compared is compared with the first input information, and the similarity of the sound input is obtained, A program that obtains the result of the comparison using the similarity of sound entry. The feature amount to be compared in the comparison step further includes information on a change in pitch,
The first feature amount extraction step includes:
From the voice acquired in the voice acquisition step, further comprising a first pitch change information acquisition step of acquiring first pitch change information that is a set of frequencies having the strongest amplitude per unit time,
The comparison step includes
Compare the second pitch change information and the first pitch change information, which is a set of frequencies of the strongest amplitude for each unit time of the audio data to be compared, and obtain the similarity regarding the pitch change, The program according to claim 9, wherein the comparison result is acquired using a similarity degree related to the change in the pitch. The feature amount to be compared in the comparison step further includes information on vibrato,
The first feature amount extraction step includes:
A first vibrato information acquisition step of acquiring first vibrato information that is a set of frequencies having the strongest amplitude per unit time from the voice acquired in the voice acquisition step;
The comparison step includes
It is a set of frequencies acquired from the audio data to be compared, and the second vibrato information, which is a set of frequencies having the strongest amplitude per unit time, is compared with the first vibrato information, and similarities related to vibrato The program according to claim 9 or 10, wherein a degree is obtained, and the result of the comparison is obtained using the degree of similarity related to the vibrato. 12. The program according to claim 9, wherein the feature amount compared in the comparison step does not include information on tempo. The unit time is
The program according to any one of claims 9 to 12, which is approximately 0.03 seconds.

Images