JP2017111274A - Data processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily utilize sound data by adjusting a reproduction timing of acquired sound data.SOLUTION: A data processor comprises; a sound data acquisition part for acquiring first sound data including time information; a featured value calculation part for generating first featured value data associated with the time information; a specification part for, on the basis of second featured value data and the first featured value data, specifying a first data position of the first featured value data corresponding to a second data position of the second featured value data; a moving part for, on the basis of a temporal positional relation between the first data position and the second data position, changing the time information such that reproduction timings move in parallel; and an expansion/contraction part for comparing the first featured value data and the second featured value data associated with the changed time information to detect a correspondence relation between respective data portions, and for changing the time information such that an interval between the reproduction timings expands/contracts on the basis of the correspondence relation.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a technique for adjusting a plurality of data.

  In recent karaoke apparatuses, not only singing but also various functions are added. Examples include a function for analyzing and evaluating a singing voice, a function for automatically adjusting the tempo of a song, and the like. A technique for automatically adjusting the tempo of a song according to the singing is disclosed in Patent Document 1, for example. Patent Document 2 discloses a technique for synchronized performance of a plurality of parts.

Japanese Patent No. 3944930 JP 2003-263161 A

  An object of the present invention is to facilitate the utilization of sound data by adjusting the reproduction timing of acquired sound data.

  According to an embodiment of the present invention, the sound data acquisition unit that acquires the first sound data including time information that defines the reproduction timing of each part of the data, and the feature amount of the first sound data are calculated in time series. , A feature amount calculation unit that generates first feature amount data in which the feature amount is associated with the time information, second feature amount data indicating a feature amount of second sound data, and the first feature A specifying unit that specifies a first data position of the first feature amount data corresponding to a second data position of the second feature amount data based on the amount data; the first data position; Based on a temporal positional relationship with the second data position, a moving unit that changes the time information so as to translate the reproduction timing, and the first information associated with the changed time information Feature quantity data and the second feature quantity And a data expansion / contraction unit that detects a correspondence relationship between the respective data by comparing with the data and changes the time information so as to expand and contract the reproduction timing interval based on the correspondence relationship. A processing device is provided.

  The expansion / contraction unit moves the predetermined reproduction timing of the first sound data based on the correspondence relationship, so that the reproduction timing of one of the first interval before the reproduction timing and the second interval after the reproduction timing is changed. The time information may be changed based on the expansion / contraction process including an expansion / contraction process for extending the interval and reducing the reproduction timing interval on the other hand.

  The expansion / contraction unit may change the time information by dividing the range of the sound data into a plurality of divided sections and performing a divided expansion / contraction process for performing the expansion / contraction process in the divided section.

  The expansion / contraction part may execute the division expansion / contraction process a plurality of times, and each time the division expansion / contraction process is executed, the division section may be short.

  The expansion / contraction part may change the time information so as to expand / contract with respect to the entire first sound data based on the correspondence relationship before the expansion / contraction process.

  The second sound data includes a plurality of sound data, the second feature amount data indicates a feature amount of the plurality of sound data and a distribution of the feature amount, and the expansion / contraction part includes the second sound data. The correspondence relationship may be detected based on the distribution of the feature amount in the feature amount data.

  You may further provide the synthetic | combination part which synthesize | combines the said 1st sound data containing the said time information changed by the said expansion-contraction part with the said 2nd sound data.

  According to one embodiment of the present invention, the computer acquires first sound data including time information that defines the reproduction timing of each part of the data, and calculates a feature amount of the first sound data in time series. First feature value data in which the feature value is associated with the time information is generated, and based on the second feature value data indicating the feature value of the second sound data and the first feature value data , Specifying a first data position of the first feature quantity data corresponding to a second data position of the second feature quantity data, and a time between the first data position and the second data position The time information is changed so as to translate the reproduction timing based on a specific positional relationship, and the first feature amount data and the second feature amount data associated with the changed time information And compare each part of the data Detecting a relationship, a program for executing to change the time information so as to expand and contract the distance between the reproduction timing based on the correspondence relationship is provided.

  According to an embodiment of the present invention, it is possible to facilitate the utilization of sound data by adjusting the reproduction timing of the acquired sound data.

It is a block diagram which shows the structure of the data processing system in 1st Embodiment of this invention. It is a block diagram which shows the structure of the evaluation apparatus in 1st Embodiment of this invention. It is a block diagram which shows the structure of the evaluation function in 1st Embodiment of this invention. It is a block diagram which shows the structure of the expansion-contraction function in 1st Embodiment of this invention. It is a flowchart which shows the expansion-contraction process in 1st Embodiment of this invention. It is a figure which shows the specific example of the specific process and movement process in 1st Embodiment of this invention. It is a figure which shows the specific example of the whole expansion-contraction process in 1st Embodiment of this invention. It is a figure which shows the specific example of the 1st division | segmentation expansion / contraction process in 1st Embodiment of this invention. It is a figure which shows the specific example of the 2nd division expansion / contraction process in 1st Embodiment of this invention. It is a figure which shows the specific example (example when only a part of music is sung) of the specific process and movement process in 1st Embodiment of this invention. It is a figure which shows the specific example (example in case the comparison object of a song is a guide melody) in the specific process and movement process in 2nd Embodiment of this invention. It is a flowchart which shows the expansion-contraction process in 3rd Embodiment of this invention. It is a figure which shows the example of the evaluation object window in 3rd Embodiment of this invention. It is a figure explaining OFFSET calculated by the OFFSET calculation process in 3rd Embodiment of this invention. It is a figure explaining SHIFT calculated by the SHIFT calculation process in 3rd Embodiment of this invention. It is a figure explaining the correspondence of each data part of model feature-value data Sd and singing feature-value data Ss in 3rd Embodiment of this invention. It is a figure explaining expansion and contraction of the song feature-value data in 3rd Embodiment of this invention.

  Hereinafter, a data processing system according to an embodiment of the present invention will be described in detail with reference to the drawings. The following embodiments are examples of the embodiments of the present invention, and the present invention is not limited to these embodiments. Note that in the drawings referred to in the present embodiment, the same portion or a portion having a similar function is denoted by the same reference symbol or a similar reference symbol (a reference symbol simply including A, B, etc. after a number) and repeated. The description of may be omitted.

<First Embodiment>
[Overview]
The data processing system according to the first embodiment of the present invention will be described in detail with reference to the drawings. The data processing system in the first embodiment registers singing voice data (singing voice data) of a user who sings (hereinafter may be called a singer) in a database. The singing voice registered in the database is statistically processed, and is used, for example, as a reference for karaoke singing evaluation. According to this evaluation standard, singing evaluation with respect to the average singing of a singer can be performed. Moreover, a duet song can also be reproduced by playing back using a part (for example, two) of the singing voices registered in the database.

  Depending on the content of the song, there may be a time lag (early song or late song) between the song already registered in the database and the song to be registered. In this way, when there is a time lag, even if the music is the same, the parts sung at the same playback timing will be different. Even if singing voice data is collected in such a state, it cannot be said to be useful data and a set.

  Therefore, the data processing system according to the first embodiment expands and contracts the singing on the time axis by changing the information defining the reproduction timing of each part of the data in the singing voice data registered in the database. Hereinafter, “extension / contraction” refers to expansion / contraction on the time axis, but depending on the expansion / contraction processing, the singing pitch is maintained and is not changed. With this processing, the data already registered and the data registered from now on are adjusted so that the same portion is sung as much as possible at the same reproduction timing. The data processing system in the first embodiment will be described below.

[Data processing system configuration]
FIG. 1 is a block diagram showing a configuration of a data processing system according to the first embodiment of the present invention. The data processing system 1000 includes an evaluation device 1, a data processing device 3, and a database 5. Each of these components is connected via a network NW such as the Internet. In this example, a plurality of evaluation devices 1 are connected to the network NW. The evaluation device 1 is, for example, a karaoke device, and in this example, is a karaoke device capable of singing evaluation. In addition, portable apparatuses, such as a smart phone, may be sufficient.

  In this example, the singing voice is input in these evaluation devices 1, the singing voice is expanded and contracted in the data processing device 3, and the expanded and contracted singing voice data is registered in the database 5. In the database 5, the singing voice data generated in each evaluation device 1 is converted so that the singing voice is expanded and contracted, and is registered in association with each music piece.

  Here, not only the singing voice data but also the feature quantity data obtained by calculating a predetermined feature quantity from the singing voice data may be registered in the database 5. In this example, the feature amount corresponds to the pitch of the singing voice. It is registered as feature data in association with the data of each singing voice. The feature amount may be calculated by the evaluation device 1 or may be calculated by the data processing device 3 or the database 5. In this example, a plurality of singing voice data associated with each piece of music is used as an evaluation criterion when the singing voice is evaluated in the evaluation device 1. This evaluation standard is based on data (model voice data) as a model voice that summarizes a plurality of singing voices, and data obtained by statistically processing a plurality of singing voices (for example, pitch distribution at each timing, etc. Distribution data). In this example, feature amount data (exemplary feature amount data) representing a temporal change in the feature amount of the model voice (pitch of the model voice in this example) is further included. Here, the model feature value data is data of an average value or a median value of feature values for a plurality of singing voices. This data may include distribution data.

  The model voice data and the singing voice data include time information for defining the reproduction timing of each part (data position) of the data by the time code. As the time code, for example, an SMPTE code that defines the reproduction timing in hours, minutes, seconds, and frames may be used. The time information may be any information as long as it defines the reproduction timing in each part of the data. Note that the distribution data and the exemplary feature data also include time information corresponding to the exemplary audio data. Although the distribution data and the feature amount data are not reproduced, the time information functions as information for specifying the data position in time series in association with the model voice data.

[Data processor configuration]
As shown in FIG. 1, the data processing device 3 includes a control unit 31, a storage unit 33, and a communication unit 39. The control unit 31 includes an arithmetic processing circuit such as a CPU. The control unit 31 executes a control program stored in the storage unit 33 by the CPU, and realizes various functions in the data processing device 3. The realized function includes a function of expanding and contracting the singing voice in time (sometimes referred to as an expansion / contraction function). The expansion / contraction function will be described later.

  The storage unit 33 is a storage device such as a nonvolatile memory or a hard disk. The storage unit 33 stores a control program for realizing the expansion / contraction function. The control program only needs to be executable by a computer, and may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the data processing device 3 only needs to include a device that reads the recording medium. Further, the control program may be downloaded via the network NW. Based on the control of the control unit 31, the communication unit 39 connects to the network NW and transmits / receives information to / from an external device connected to the network NW.

[Configuration of evaluation device]
FIG. 2 is a block diagram showing a configuration of the evaluation apparatus according to the first embodiment of the present invention. The evaluation device 1 includes a control unit 11, a storage unit 13, an operation unit 15, a display unit 17, a communication unit 19, and a signal processing unit 21. Each of these components is connected via a bus. In addition, a microphone 23 and a speaker 25 are connected to the signal processing unit 21.

  The control unit 11 includes an arithmetic processing circuit such as a CPU. The control unit 11 causes the CPU to execute the control program stored in the storage unit 13 to realize various functions in the evaluation device 1. The realized functions include a function for evaluating the singing voice (hereinafter sometimes referred to as an evaluation function). The storage unit 13 is a storage device such as a nonvolatile memory or a hard disk. The storage unit 13 stores a control program for realizing the evaluation function. The control program only needs to be executable by a computer, and may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. In this case, the evaluation device 1 only needs to include a device that reads the recording medium. Further, the control program may be downloaded via the network NW.

  Moreover, the memory | storage part 13 memorize | stores music data, song audio | voice data, and evaluation criteria information as data regarding a song. The music data includes data related to the karaoke song, for example, guide melody data, accompaniment data, and lyrics data. The guide melody data is data indicating the melody of the song. Accompaniment data is data indicating the accompaniment of a song. The guide melody data and accompaniment data may be data expressed in the MIDI format. The lyric data is data for displaying the lyrics of the song and data indicating the timing for changing the color of the displayed lyrics telop. These data may be acquired from an external server.

  The singing voice data is data indicating the singing voice input from the microphone 23 by the singer and is buffered in the storage unit 13. The singing voice data includes the time information described above. The evaluation reference information is information used by the evaluation function as a reference for evaluating the singing voice, and includes, for example, information on an arithmetic expression for calculating an evaluation value. In addition, the song used as the reference | standard when compared with the singing voice | voice of evaluation object, ie, model singing (model voice), may be contained in evaluation criteria information. The evaluation criterion information may be generated by acquiring the evaluation criterion registered in the database 5.

  The operation unit 15 is a device such as an operation button, a keyboard, or a mouse provided on an operation panel and a remote controller, and outputs a signal corresponding to the input operation to the control unit 11. With this operation unit 15, for example, an input operation generally performed in a karaoke apparatus such as selection of music can be performed. The display unit 17 is a display device such as a liquid crystal display or an organic EL display, and displays a screen based on control by the control unit 11. Note that the operation unit 15 and the display unit 17 may integrally form a touch panel. The communication unit 19 is connected to a communication line such as the Internet based on the control of the control unit 11 and transmits / receives information to / from an external device such as a server. The function of the storage unit 13 may be realized by an external device that can communicate with the communication unit 19.

  The signal processing unit 21 includes a sound source that generates an audio signal from a MIDI format signal, an A / D converter, a D / A converter, and the like. The singing voice is converted into an electric signal by the microphone 23 and input to the signal processing unit 21, and A / D converted by the signal processing unit 21 and output to the control unit 11. As described above, the singing voice is buffered in the storage unit 13 as singing voice data. The accompaniment data is read out by the control unit 11, D / A converted by the signal processing unit 21, and output from the speaker 25 as an accompaniment of the song. At this time, a guide melody may be output from the speaker 25.

[Evaluation function]
An evaluation function realized by the control unit 11 of the evaluation apparatus 1 executing a control program will be described. A part or all of the configuration for realizing the evaluation function described below may be realized by hardware.

  FIG. 3 is a block diagram showing the configuration of the evaluation function in the first embodiment of the present invention. The evaluation function 100 includes an input sound acquisition unit 101, a feature amount calculation unit 103, a comparison unit 105, and an evaluation value calculation unit 107. The input sound acquisition unit 103 acquires singing voice data indicating the singing voice input from the microphone 23.

  The feature amount calculation unit 103 analyzes the singing voice data acquired by the input sound acquisition unit 101, and calculates a temporal change in the singing feature amount, in this example, the pitch (frequency) of the singing voice, that is, the singing pitch waveform. To do. Specifically, the singing pitch waveform is calculated by a known method such as a method using a zero cross of the waveform of the singing voice or a method using FFT (Fast Fourier Transform). In addition to the pitch, the feature amount may be a volume level, a harmonic ratio, or the like. The feature amount is not limited to a value obtained as the characteristic value itself, but may be a value obtained as a change amount of the characteristic value by an operation using differentiation or the like. Since the absolute value component can be removed in this way by the amount of change, for example, a key-shifted singing can be appropriately evaluated.

  The comparison unit 105 compares the feature amount (singing pitch waveform) of the singing voice obtained by the feature amount calculation unit 103 with the feature amount (exemplary pitch waveform) of the model voice. The comparison unit 105 acquires model feature data registered in the database 5 in association with the sung song, and obtains model voice feature data based on the model feature data. As a comparison result, in this example, the degree of coincidence between the singing pitch waveform and the exemplary pitch waveform is calculated based on, for example, the distance between the waveforms. Here, the closer the distance between the waveforms, the higher the degree of coincidence. At this time, by obtaining distribution data also from the database 5, the calculation of the degree of coincidence may be weighted in consideration of the pitch distribution of the model voice. For example, even if the pitch of the singing sound and the model voice in the comparison target part is far apart, if the pitch distribution of the model voice is wide, the degree of coincidence is less likely to be lower than when the distribution is narrow. Good.

  The evaluation value calculation unit 107 calculates an evaluation value that serves as an index for evaluating the singing voice based on the comparison result in the comparison unit 105. In this example, the higher the degree of coincidence calculated by the comparison unit 105, the higher the evaluation value is calculated, and the higher the evaluation of the singing voice. Note that the evaluation value calculation unit 107 may calculate the evaluation value based not only on the degree of coincidence but also based on other factors. This evaluation result may be presented on the display unit 17. The above is the description of the evaluation function 100.

[Extension function]
The expansion / contraction function realized by the control unit 31 of the data processing device 3 executing the control program will be described. Note that part or all of the configuration for realizing the expansion / contraction function described below may be realized by hardware.

  FIG. 4 is a block diagram showing the configuration of the expansion / contraction function in the first embodiment of the present invention. The expansion / contraction function 300 includes a sound data acquisition unit 301, a feature amount calculation unit 303, a specification unit 305, a movement unit 307, and an expansion / contraction unit 309.

  The sound data acquisition unit 301 acquires the singing voice data generated in the evaluation device 1 via the network NW. This singing voice data includes time information for defining the reproduction timing of each part of the data as described above.

  The feature amount calculation unit 303 analyzes the singing voice data acquired by the sound data acquisition unit 301, calculates the feature amount of the song in time series, and generates feature amount data indicating the feature amount. The feature amount calculation method is the same as the method in the feature amount calculation unit 103 described above. That is, the feature amount is the pitch of the singing voice, and the feature amount data represents a temporal change of this pitch. The generated feature data includes the time information described above. That is, the feature quantity of the singing voice at a specific playback timing in the singing voice data corresponds to the feature quantity at the data position corresponding to the specific playback timing in the feature quantity data.

  The specifying unit 305 is based on the feature amount data (referred to as singing feature amount data) generated in the feature amount calculating unit 303 and the model feature amount data registered in the database 5 in association with the sung song. Among the data, the relationship about the predetermined data position is specified. In this example, the predetermined data position indicates the data position at the beginning of the singing section. Therefore, the specifying unit 305 specifies the data position (referred to as the second data position) in the model feature data corresponding to the beginning data position (referred to as the first data position) of the singing section in the singing feature value data. A specific example of the processing (expansion / contraction processing) of the specifying unit 305, the moving unit 307, and the expansion / contraction unit 309 will be described later.

  The moving unit 307 changes the time information so as to translate the playback timing of the singing voice data based on the data position relationship specified by the specifying unit 305. The time information changed in this way is time information included in the singing voice data and the singing feature amount data. Here, the first data position of the singing feature value data coincides with the second data position of the model feature value data by shifting the time code of the time information as a whole, that is, by moving the data position in parallel. Change the time information as follows.

  The expansion / contraction unit 309 compares the singing feature amount data whose time information has been changed by the moving unit 307 with the model feature amount data, and detects the correspondence between the data units. This correspondence relationship indicates the relationship between the data portions where the similarity between the singing feature amount data and the model feature amount data is high. A specific method for detecting the correspondence will be described in a specific example described later. The expansion / contraction unit 309 changes the time information so as to expand and contract the data interval (reproduction timing interval) based on this correspondence. The time information changed here is also the time information included in the singing voice data and the singing feature amount data. The expansion / contraction unit 309 registers the singing voice data and the singing feature amount data whose time information is changed in the database 5 in association with the sung music. The above is the description of the expansion / contraction function 300.

[Extension processing]
Next, the expansion / contraction process executed in the expansion / contraction function 300 will be specifically described. The expansion / contraction process is a collection of processes executed in the specifying unit 305, the moving unit 307, and the expansion / contraction unit 309 in the expansion / contraction function 300 described above. Therefore, the expansion / contraction process is started when the singing feature amount data is calculated by the feature amount calculation unit 303.

  FIG. 5 is a flowchart showing the expansion / contraction processing in the first embodiment of the present invention. FIG. 6 is a diagram showing a specific example of the specifying process and the moving process in the first embodiment of the present invention. FIG. 7 is a diagram showing a specific example of the entire expansion / contraction process in the first embodiment of the present invention. FIG. 8 is a diagram showing a specific example of the first division expansion / contraction process in the first embodiment of the present invention. FIG. 9 is a diagram showing a specific example of the second divided expansion / contraction process in the first embodiment of the present invention. The processing of steps S101 and S103 described below is executed by the specifying unit 305, and the processing of step S105 is executed by the moving unit 307. These processes will be described with reference to FIG. Further, the processing of steps 107 to S111 is executed by the expansion / contraction unit 309. These processes will be described with reference to FIGS.

  FIG. 6 shows a diagram simulating the section Dd in which the model voice data exists, the model feature data Sd, the section Ds in which the singing voice data exists, and the singing feature data Ss. The horizontal axis direction is the time axis t. First, the singing section Ds1 is extracted based on the singing feature amount data Ss (FIG. 5; step S101). In this example, the singing section Ds1 is defined as a section corresponding to an area where a feature amount (pitch) can be calculated by using a so-called SAD (song activity direction) method. The so-called VAD (voice activity direction) method may be used to determine the singing section Ds1 according to the volume level.

  Subsequently, a data position corresponding to a predetermined data position of the singing feature value data Ss is specified in the model feature value data Sd (FIG. 5; step S103). As described above, the predetermined data position here indicates the data position at the beginning of the singing section. Of the singing feature amount data Ss, the data portion in the beginning section Ds2 of the singing section Ds1 is compared with the model feature amount data Sd. Based on the comparison result, the section Dd2 similar to the data portion of the section Ds2 is detected from the exemplary feature data Sd.

  In this example, the waveform of the singing feature value data Ss in the section Ds2 is compared with the waveforms of various regions in the model feature data Sd, and the region having the highest similarity is specified as the section Dd2. Accordingly, the data position of the section Ds2 (for example, the reproduction timing ts1 of the center data position of the section Ds2) in the singing feature value data Ss and the data position (for example, of the section Dd2 of the model feature data Sd). It is specified that the reproduction timing td1) at the center data position has a corresponding relationship.

  The degree of similarity is calculated in the same manner as the degree of coincidence calculated in the comparison unit 105 described above. That is, the closer the distance between the waveforms is, the higher the similarity is calculated. Note that distribution data corresponding to the exemplary feature quantity data may be acquired from the database 5 and the feature quantity distribution may be used to calculate the similarity. For example, only the waveform of the exemplary feature quantity data in a section where the distribution of the feature quantity has a small spread, that is, a section where the variation of the feature quantity is small may be the target of similarity calculation. Moreover, you may weight the calculation of similarity according to distribution. Note that the similarity may be calculated as the evaluation value by a method similar to the evaluation value calculation algorithm in the evaluation apparatus 1.

  Subsequently, the singing feature value data Ss is translated on the time axis based on the data positions in the correspondence relationship (FIG. 5; step S105). When moving in parallel, the time information of the singing feature amount data Ss is changed so that the reproduction timing ts1 becomes the reproduction timing td1. That is, it may be changed so that (td1-ts1) is added to the reproduction timing of each data position. The data in which the time information is changed in this way is singing feature amount data Ss2 shown in FIG.

  Subsequently, the singing feature value data Ss2 and the model feature value data Sd are compared, and the entire data of the singing feature value data Ss2 is expanded or contracted (FIG. 5; step S107). This process is referred to as an entire expansion / contraction process. Here, the entire time data of the singing feature value data Ss2 is proportionally expanded and contracted, and the time information is changed so that the similarity to the model feature value data Sd is as high as possible. At this time, the data position of the reproduction timing ts1 or the first data position of the section Ds2 is not changed. For example, in the case of reducing to 90% as a whole, time information may be obtained by multiplying the reproduction timing of the data position by 0.9 for the data after the data whose position is not changed. Thus, when expanding and contracting, the time information is changed so as to expand and contract evenly in the section to be expanded and contracted. The split expansion / contraction process described below is also similar in that the time information is changed so that expansion / contraction within the same section is evenly expanded / contracted.

  The data in which the time information is changed is singing feature amount data Ss3 shown in FIG. The calculation of similarity here may be performed in the same manner as described above. For example, the singing feature value data Ss2 is expanded and contracted so that the distance between the waveforms of the singing feature value data Ss3 and the model feature value data Sd is as small as possible. In calculating the similarity, the distribution data may be included in the calculation as described above. In FIG. 7, the playback timing of the last data position in the singing section is shown to match, but the playback timing of the last data position depends on the similarity as described above, and is not necessarily shown in the drawing. Not necessarily street.

  Subsequently, the singing feature quantity data Ss3 is divided into a plurality of sections, the model feature quantity data Sd is compared with each section, and the singing feature quantity data Ss3 is expanded or contracted for each section (FIG. 5; step S109). . This process is called a division expansion / contraction process. The division expansion / contraction process is desirably performed a plurality of times. First, the first division expansion / contraction process will be described with reference to FIG. A midpoint data position M1 between the first data position ST and the last data position EN is determined. Thus, the singing feature amount data Ss3 is divided into the section SP1 between ST and M1, and the section SL1 between M1 and EN.

  The expansion / contraction of the data here is executed by moving the reproduction timing of the data point M1 at the middle point back and forth in time. When M1 is moved forward in time, the section SP1 is reduced and the section SL1 is expanded. Conversely, when M1 is moved later in time, the section SP1 is expanded and the section SL1 is contracted. At this time, the data positions of ST and EN corresponding to one end of each section are not changed. Therefore, ST and EN may be referred to as fixed delimiter positions. M1 may be referred to as a movement delimiter position.

  As shown in FIG. 8, when the data position M1 is moved to M1s at an earlier time, the section SP1 is reduced to SP1s and the section SL1 is extended to SL1s. The singing feature value data Ss4 is obtained by changing the time information so as to expand and contract the singing feature value data Ss3. The reproduction timing of M1s, which is the movement destination of M1, is determined so that the similarity between the singing feature value data Ss4 and the model feature value data Sd is as high as possible. The calculation of the similarity may be performed by the method described above.

  When the division expansion / contraction process shown in FIG. 8 ends, it is determined whether the division unit has reached the minimum value (FIG. 5; step S111). The division unit refers to one of the divided sections in the example of FIG. When the section length is equal to or smaller than a predetermined minimum value (step S111; Yes), the expansion / contraction process is terminated. On the other hand, when the minimum value has not been reached (step S111; No), the division expansion / contraction process is executed again (step S109).

  When the division expansion / contraction process in step S109 is executed after the second time, the division unit at the time of the division expansion / contraction process performed immediately before is made finer each time it is executed. In this example, as shown in FIG. 9, the division unit is made finer than the first time as four divisions of sections SP1, SL1, SP2, and SL2. First, the singing feature value data Ss4 is roughly divided into two. Here, it is divided into two sections, a section from ST1 to EN1, and a section from ST2 to EN2. EN1 (ST2) is the midpoint between ST1 and EN2. ST1, EN1 (ST2), and EN2 are delimitation positions (fixed delimitation positions) at which data positions are fixed in the division expansion / contraction process. On the other hand, M1 and M2 are set as delimiter positions (movement delimiter positions) for moving the data positions at the midpoints of the two sections. As a result, it is divided into the above four sections.

  The division expansion / contraction process at this time is the same as the first division expansion / contraction process by moving M1 back and forth in time to expand / contract sections SP1 and SL1, and moving M2 back and forth in time. Expand and contract SP2 and SL2. If M1 is moved to M1s and M2 is moved to M2s as in the example of FIG. 9, sections SP1 and SP2 are expanded to SP1s and SP2s, respectively, and sections SL1 and SL2 are reduced to SL1s and SL2s, respectively. The The singing feature value data Ss5 is obtained by changing the time information so as to expand and contract the singing feature value data Ss3. The data positions of M1s and M2s, which are the movement destinations of M1 and M2, are determined so that the similarity between the singing feature value data Ss5 and the model feature value data Sd is as high as possible.

  In the third division expansion / contraction process, the number of divisions is further increased to make the division unit finer. At this time, ST1 to ST4 and EN1 to EN4 are determined as fixed delimiter positions. Also, M1, M2, M3, and M4 are defined as movement partition positions between adjacent fixed partition positions. The division unit (sections SP1, SL1,..., SP4, SL4) is determined by the fixed division position and the movement division position. In the same manner as described above, the movement delimiter positions M1, M2,... Are moved to expand and contract each section.

  In this way, the division expansion / contraction process is repeated while making the division unit fine until the division unit reaches a predetermined minimum value. The time information is changed by repeating the division expansion / contraction process. Accordingly, the singing feature value data is expanded and contracted so as to be similar to the model feature value data. Along with this, the changed time information is also applied to the singing voice data, and the singing voice data is expanded and contracted so as to be similar to the model voice data.

  The division expansion / contraction process can be variously changed. For example, when the number of division expansion / contraction processes increases, the number of divisions is increased to make the division unit fine, that is, to shorten one section length, but according to the number of processes according to a predetermined rule. The number of divisions may be increased or decreased. In addition, the singing feature amount data is equally divided so that the respective segment lengths after the division are the same, but may be divided so that the respective segment lengths are different according to a predetermined rule. . In step S111, it is not determined that the division unit has reached the minimum value, but it is determined that the similarity between the singing feature value data and the model feature value data after the division expansion / contraction processing is equal to or greater than a predetermined value. You may make it do.

  In the above example, the case where the entire music is sung has been described. On the other hand, even when only a part of the song is sung or the like, when the singing voice data with a short singing period is expanded and contracted, according to the expansion / contraction processing, the processing can be performed as shown in FIG. is there.

  FIG. 10 is a diagram showing a specific example of the specifying process and the moving process in the first embodiment of the present invention (an example in the case where only a part of a song is sung). As shown in FIG. 10, the section Ds in which the singing voice data exists is significantly shorter than the section Dd in which the model voice data exists. In such a case, only a part of the music may be sung. Even in this case, the same process as the expansion / contraction process shown in FIG. As a result, processing is performed as in the example shown in FIG. 10 in the same manner as in the example shown in FIG. That is, the singing section Ds1 is extracted, the waveform of the singing feature value data Ss of the opening section Ds2 is compared with the waveforms of various regions of the model feature value data Sd, and the region having the highest similarity is defined as the section Dd2. Identify.

  When there are two choruses of music, the sung section may be any one chorus. At this time, it may be difficult to determine whether the first chorus is similar or the second chorus is similar. In such a case (when there is a section where the difference in similarity is within a predetermined value), another type of feature amount may be further used. For example, in order to determine the difference in phonemes due to the difference in lyrics, it may be desirable to use a harmonic ratio.

  As a result, the reproduction timing ts2 of the predetermined data position in the section Ds2 in the singing feature value data Ss and the reproduction timing td2 in the predetermined data position of the section Dd2 in the model feature data Sd are identified as corresponding to each other. To do. Then, based on the data position in the correspondence relationship, the time information is changed so that the reproduction timing ts2 becomes the reproduction timing td2, and the singing feature value data Ss is translated on the time axis. In the example shown in FIG. 10, the time information is changed so as to move to the section SA. Subsequent expansion / contraction processing is performed within the range of the section SA. In the division expansion / contraction process, the section SA may be divided into a plurality of sections and expanded / contracted.

  In addition, although the whole music is sung, the distance between the first chorus and the second chorus is long, and the singing section may be detected in a plurality of parts. In such a case, the process shown in FIG. 10 may be performed for each singing section to perform the expansion / contraction process.

  As described above, in the data processing system 1000 according to the first embodiment, the singing voice data of the singer input in the evaluation device 1 is registered in the database 5. At this time, even if there is a time lag (early singing or late singing) from the singing voice data already registered as a part of the model voice data in the database 5 as the same music, this deviation is reduced. be able to. As a result, new singing voice data can be included in a part of the model sound data, and can be made useful data when statistically processing the singing feature amount. Moreover, when it reproduces | regenerates using two song audio | speech data among several song audio | voice data of the same music registered into the database 5, a duet song can also be reproduced. These singing voice data can also be downloaded and listened to the evaluation apparatus 1 or various terminals (smartphone, personal computer, etc.) via the network NW.

Second Embodiment
In the second embodiment, a case will be described in which model feature data that is a comparison target when expanding and contracting singing voice data is data indicating a guide melody. As described above, the exemplary feature amount data is not limited to data indicating the waveform of the feature amount such as the pitch of the singing voice, but may be data representing a melody with a discrete pitch (for example, a pitch determined in units of 100 cents). Good. A specific example in this case will be described with reference to FIG.

  FIG. 11 is a diagram showing a specific example of the specifying process and the moving process in the second embodiment of the present invention. As shown in FIG. 11, the exemplary feature data Sd is data representing a melody by a discrete pitch and its period. The example which expands / contracts the singing feature-value data Ss similar to said embodiment on the basis of such model feature-value data Sd is shown. Similar to the expansion / contraction processing shown in FIG. 5, the singing section Ds1 is extracted, and the waveform of the singing feature amount data Ss in the opening section Ds2 is compared with the melody of various regions of the model feature amount data Sd, and the degree of similarity is highest. A region in which the value becomes high is specified as the section Dd2. For calculating the similarity, for example, a melody with a discrete pitch is regarded as a waveform, and the distance from the waveform of the singing feature amount data Ss in the section Ds2 may be used as in the above embodiment.

  As a result, the reproduction timing ts3 of the predetermined data position in the section Ds2 in the singing feature value data Ss and the reproduction timing td3 in the predetermined data position of the section Dd2 in the model feature data Sd are identified as corresponding to each other. To do. Then, based on the data position in the correspondence relationship, the time information is changed so that the reproduction timing ts3 becomes the reproduction timing td3, and the singing feature value data Ss is translated on the time axis. The subsequent expansion / contraction process is executed in the same manner as in the first embodiment. The difference from the first embodiment is that the model to be compared is a melody having a discrete pitch. Similar to the above, the similarity can be calculated by considering a melody having a discrete pitch as a waveform and using the distance from the waveform of the singing feature value data as in the above embodiment.

<Third Embodiment>
In the third embodiment, the expansion / contraction process is performed by a method different from that of the first embodiment. However, the identifying unit 305 and the moving unit 307 perform substantially the same processing as in the first embodiment. On the other hand, the expansion / contraction unit 309 changes the time information according to the distance between the waveforms of the singing feature value data and the model feature value data, but the process is different from the first embodiment. . That is, the division expansion / contraction process is used in the first embodiment, but the sequential expansion / contraction process is adopted in the third embodiment. The sequential expansion / contraction process will be described below.

  FIG. 12 is a flowchart showing expansion / contraction processing in the third embodiment of the present invention. A singing section Ds is extracted based on the singing feature amount data Ss (FIG. 12; step S201). In this example, the model feature value data Sd is data in which the singing section Dd is extracted in advance. Subsequently, OFFSET is calculated (step S203). This process is executed by the specifying unit 305. The OFFSET calculation method will be described with reference to FIGS.

  FIG. 13 is a diagram illustrating an example of an evaluation target window in the third embodiment of the present invention. FIG. 14 is a diagram for explaining OFFSET calculated by the OFFSET calculation processing in the third embodiment of the present invention. First, as shown in FIG. 13, a section Bd is set at the beginning of the model feature data Sd, and a section Bs is set at the beginning of the singing feature data Ss. Here, the section Bd and the section Bs have the same length, for example, a width of about 30 seconds.

  Subsequently, as shown in FIG. 14, the singing feature value data Ss (singing section Ds) is translated in time as a whole, and the data portion of the section Bd of the model feature value data Sd and the singing feature value data Ss. Are compared with the data portion of the section Bs, and the position of the section Bs having the highest similarity is calculated. The degree of similarity corresponds to the distance between the waveforms of the data portion of the section Bd of the exemplary feature data Sd and the data portion of the section Bs of the singing feature data Ss. Then, the point with the highest similarity is calculated using the method of least squares.

Specifically, it is calculated as follows. A case where the waveform of the exemplary feature data Sd is expressed as a function of time t is defined as DSd (t). In addition, a case where the waveform of the singing feature value data Ss is expressed as a function of time t is DSs (t). And let ts be the parallel movement amount of the singing feature amount data Ss. ΣC is calculated by adding C = (DSd (t) −DSs (t + ts)) ² in the range t of the section Bd (for example, t = 0 to tb). Then, ts where ΣC is the smallest is detected. The detected ts is the OFFSET value to be calculated. In this way, OFFSET is calculated. As described in the first embodiment, when the section Ds is short, only a part of the section Dd may be sung. In this case, OFFSET may be a large value. Even in this case, the subsequent processing can be performed in the same manner.

  Subsequently, the singing feature value data Ss is translated on the time axis (FIG. 12; step S205). When translating, the time information of the singing feature value data Ss is changed so that the calculated OFFSET is applied. That is, the time information may be changed so that the reproduction timing of each data position of the singing feature value data Ss is shifted by OFFSET. In the following description, the singing feature amount data Ss1 (singing section Ds1) indicates data in which time information is changed by reflecting OFFSET.

  Subsequently, the SHIFT corresponding to the difference between the exemplary feature data Sd and the singing feature data Ss1 is calculated (FIG. 12; step S207). A method for calculating SHIFT will be described with reference to FIG.

  FIG. 15 is a diagram for explaining the SHIFT calculated by the SHIFT calculation process according to the third embodiment of the present invention. First, the position of the section Bd set at the beginning of the model feature data Sd is changed. The amount to be changed is a predetermined time (referred to as unit time). The unit time may be shorter or longer than the length of the section Bd.

  Then, a section Bs that is a position corresponding to the section Bd is detected. Here, the position of the section Bs in the singing feature amount data Ss1 is detected so that the similarity between the data part of the section Bd after the change and the data part of the section Bs after the change becomes the highest. A specific method for calculating the position of the section Bs corresponding to the section Bd is the same as the method for calculating OFFSET. A difference in position between the detected section Bs and the section Bd is calculated as a SHIFT value.

  As shown in FIG. 15, the section Bd is advanced by unit time, and the above processing is repeated to calculate SHIFT corresponding to each section Bd (step S209; No, step S207). When the section Bd reaches the last data of the exemplary feature data Sd (step S209; Yes), the calculation of SHIFT ends. Note that SHIFT> 0 is set when the section Bs advances in time with respect to the section Bd.

  FIG. 16 is a diagram for explaining a correspondence relationship between each part of the model feature data Sd and the singing feature data Ss in the third embodiment of the present invention. The horizontal axis corresponds to the model feature data Sd, and the vertical axis corresponds to the singing feature data Ss.

  The SHIFT value corresponds to the amount of deviation from SHIFT = 0. That is, when there is no time difference between the exemplary feature value data Sd and the singing feature value data Ss other than OFFSET, this corresponds to the broken line portion of SHIFT = 0. In the section before the data position tm, the singing voice is delayed in the position to be sung with respect to the model voice, and SHIFT <0. In the section after the data position tm, the singing voice has a position where the singing voice should be sung earlier than the model voice, and SHIFT> 0. SF is the value of SHIFT in the last data. Even if there is a time lag, if the singing voice and the model voice have the same singing speed, the slope is parallel to the SHIFT = 0 slope. On the other hand, the singing voice has a slope smaller than SHIFT = 0 when the singing progress speed is slower than the model voice, and the slope higher than SHIFT = 0 when the singing progress speed is fast.

  Subsequently, based on the calculated SHIFT, a sequential expansion / contraction process for expanding / contracting the singing feature value data Ss1 is executed (step S211). In this example, the time information of the singing feature value data Ss1 is changed according to the SHIFT at each data position. For example, if the SHIFT value at each data position t is S (t), each data position “t” may be changed to “t−S (t)” in the time information of the singing feature value data Ss1. If there is a data position for which the SHIFT value is not calculated, the SHIFT value may be complemented and used. In addition, when the complexity of the line indicating the correspondence relationship between each part of the data is equal to or greater than a predetermined value, the SHIFT calculation process may be executed again. At this time, the second highest similarity may be used, or the similarity calculation method may be changed.

  The above complexity may be defined as an index indicating the degree of smoothness of the line indicating the opposing relationship of each part of the data, and as the smoothness becomes lower, the complexity is lower. For example, when there is a discontinuous point or a rapidly changing region (such as a region having a wave shape) on the line indicating the correspondence, the complexity is high.

  FIG. 17 is a diagram for explaining expansion and contraction of singing feature value data in the third embodiment of the present invention. In FIG. 17, the singing feature value data Ss (singing section Ds), the singing feature value data Ss1 (singing section Ds1) in which OFFSET is adjusted, and the singing feature value data Sss (singing section Dss) subjected to successive expansion / contraction processing, This is shown in comparison with the exemplary feature data Sd (singing section Dd).

  About singing feature-value data Ss, Ss1, and Sss, the change of the data position accompanying the change of time information is shown typically. OFFSET, tm, and SF are the same as those shown in the above example. The time information is changed so that the playback timing is advanced in the section before tm and the playback timing is returned in the section after the tm. By applying the time information thus changed to the singing voice data, the singing voice data can be stretched and changed to data similar to the model voice data.

<Fourth embodiment>
In 4th Embodiment, in the expansion-contraction function 300 of embodiment mentioned above, when registering the singing voice data and singing feature-value data which changed time information in the database 5, the process which synthesize | combines with the data already registered. The case of performing will be described.

  The expansion / contraction function in the fourth embodiment further includes a combining unit with respect to the expansion / contraction function 300 in the above embodiment. The parts other than the combining unit are the same as those of the expansion / contraction function 300, as already described. Instead of the output of the expansion / contraction unit 309 in the expansion / contraction function 300 being input to the database 5, in the fourth embodiment, the output is input to the combining unit. That is, the synthesizing unit acquires the singing voice data and the singing feature amount data in which the time information is changed in the expansion / contraction unit 309. And a synthetic | combination part acquires the model audio | voice data and model feature-value data of a music corresponding to these from the database 5, synthesize | combines with the data acquired from the expansion-contraction part 309, and updates the registration content of the database 5.

  The synthesis includes including the singing voice data acquired from the expansion / contraction unit 309 as one of a plurality of singing voice data constituting the model voice data acquired from the database 5. Similarly, the singing feature value data acquired from the expansion / contraction unit 309 is included in the model feature value data acquired from the database 5. The synthesizer 311 may further update the registered content of the database 5 by further performing statistical processing on the new model voice data and model feature data obtained by the synthesis.

<Other embodiments>
In the above embodiment, the case where the example of the feature amount is the pitch of the sound has been described. In addition to the pitch, the feature amount may be a volume level, a harmonic ratio, or the like. The feature amount is not limited to a value obtained as the characteristic value itself, but may be a value obtained as a change amount of the characteristic value by an operation using differentiation or the like. When expressed in terms of the amount of change in this way, the component of the absolute value can be removed, and for example, it is effective when the parameter does not have an important element as an absolute value such as a volume level and emphasizes relative change. . In addition, it is possible to appropriately evaluate a key-shifted song.

  The sound represented by the singing voice data and the model voice data is not limited to the voice by the singer, but may be a voice by singing synthesis or an instrument sound. If it is a musical instrument sound, it is desirable to be a single note performance.

DESCRIPTION OF SYMBOLS 1 ... Evaluation apparatus, 11 ... Control part, 13 ... Memory | storage part, 15 ... Operation part, 17 ... Display part, 19 ... Communication part, 21 ... Signal processing part, 23 ... Microphone, 25 ... Speaker, 3 ... Data processing apparatus, DESCRIPTION OF SYMBOLS 31 ... Control part, 33 ... Memory | storage part, 39 ... Communication part, 5 ... Database, 100 ... Evaluation function, 101 ... Input sound acquisition part, 102 ... Model sound acquisition part, 103 ... Feature-value calculation part, 105 ... Comparison part, DESCRIPTION OF SYMBOLS 107 ... Evaluation value calculation part, 300 ... Expansion / contraction function, 301 ... Sound data acquisition part, 303 ... Feature-value calculation part, 305 ... Specification part, 307 ... Movement part, 309 ... Expansion / contraction part

Claims (6)

A sound data acquisition unit that acquires first sound data including time information that defines the reproduction timing of each part of the data;
A feature amount calculation unit that calculates a feature amount of the first sound data in time series and generates first feature amount data in which the feature amount is associated with the time information;
Based on the second feature value data indicating the feature value of the second sound data and the first feature value data, the first feature value corresponding to the second data position of the second feature value data. A specifying unit for specifying the first data position of the data;
A moving unit that changes the time information so as to translate the reproduction timing based on a temporal positional relationship between the first data position and the second data position;
The first feature value data associated with the changed time information is compared with the second feature value data to detect the correspondence of each part of the data, and based on the correspondence, the reproduction timing An expansion / contraction part that changes the time information so as to expand / contract the interval,
A data processing apparatus comprising:   The expansion / contraction unit moves the predetermined reproduction timing of the first sound data based on the correspondence relationship, so that the reproduction timing of one of the first interval before the reproduction timing and the second interval after the reproduction timing is changed. The data processing apparatus according to claim 1, further comprising an expansion / contraction process for extending the interval and reducing the reproduction timing interval on the other hand, and changing the time information based on the expansion / contraction process.   The said expansion / contraction part divides | segments the range of the said sound data into a some division | segmentation area, and performs the division | segmentation expansion / contraction process which performs the said expansion / contraction process in the said division | segmentation area, The said time information is changed. Data processing equipment.   The data processing apparatus according to claim 3, wherein the expansion / contraction unit executes the division expansion / contraction process a plurality of times, and the division section is shortened each time the division expansion / contraction process is executed. The second sound data includes a plurality of sound data,
The second feature amount data indicates a feature amount of the plurality of sound data and a distribution of the feature amount,
5. The data processing apparatus according to claim 1, wherein the expansion / contraction unit detects the correspondence relationship based on a distribution of the feature amount in the second feature amount data.   The data processing apparatus according to claim 5, further comprising a synthesizing unit that synthesizes the first sound data including the time information changed by the expansion / contraction unit with the second sound data.

Images