JP7375302B2 - Acoustic analysis method, acoustic analysis device and program - Google Patents

Description

The present invention relates to a technique for analyzing music.

BACKGROUND ART Conventionally, techniques have been proposed for estimating various types of information from acoustics such as singing sounds or performance sounds of music. For example, Patent Document 1 discloses a configuration for estimating a code from an acoustic signal representing a song. Specifically, the key of the song is estimated from the acoustic signal, and the chord is estimated by taking the estimated key into consideration. Further, Patent Document 2 discloses a configuration for identifying the type of tonality from the shape of the power spectrum of a song. The type of tonality is specified according to the power of each note name calculated from time-series data of the power spectrum.

JP 2015-31738 Publication JP2007-248610A

In the technique disclosed in Patent Document 1, the key of a piece of music is estimated from the note that appears most frequently. However, there are some pieces of music in which the frequency of occurrence of notes corresponding to the key is low. Further, in the technique of Patent Document 2, the type of tonality is specified using the correlation between the power of each note name and the type of tonality. However, there are songs in which there is no correlation between the power of each note name and the type of tonality. That is, under the techniques of Patent Document 1 and Patent Document 2, it is actually difficult to estimate keys with high precision for a variety of songs. In consideration of the above circumstances, an object of the present invention is to estimate the key with high accuracy.

In order to solve the above problems, an acoustic analysis method according to a preferred embodiment of the present invention uses a trained model that has learned the relationship between the time series of the feature values of an acoustic signal and the key. By inputting the sequence, key information representing the key is generated.

The acoustic analysis device according to a preferred aspect of the present invention is a trained model that has learned the relationship between the time series of the feature quantities of an acoustic signal and the key, and is configured to calculate the key from the input of the time series of the feature quantities of the acoustic signal. It includes a key estimation model that generates key information to represent.

FIG. 1 is a block diagram illustrating the configuration of an acoustic analysis device according to a first embodiment. FIG. 2 is a block diagram illustrating the functional configuration of an acoustic analysis device. FIG. 3 is a schematic explanatory diagram of feature amounts and key information. FIG. 3 is an explanatory diagram of feature amounts. FIG. 3 is an explanatory diagram of key information. 3 is a flowchart illustrating a specific procedure of key estimation processing. FIG. 3 is an explanatory diagram of the operation of the learning processing section. FIG. 2 is a block diagram illustrating a functional configuration of an acoustic analysis device according to a second embodiment. FIG. 7 is an explanatory diagram of a time series of keys corrected by post-processing according to the second embodiment. 7 is a flowchart illustrating a specific procedure of post-processing according to the second embodiment. FIG. 3 is a block diagram illustrating the functional configuration of an acoustic analysis device according to a third embodiment. FIG. 7 is an explanatory diagram of a time series of keys corrected by post-processing according to the third embodiment. 12 is a flowchart illustrating a specific procedure of post-processing according to a third embodiment.

<First embodiment>
FIG. 1 is a block diagram illustrating the configuration of an acoustic analysis device 100 according to a first embodiment of the present invention. The acoustic analysis device 100 is an information processing device that estimates the key of a song by analyzing an acoustic signal V representing acoustics such as singing sounds or performance sounds of the song. The acoustic analysis device 100 selects one of the 24 types of keys from the acoustic signal V as candidates corresponding to the combination of a plurality of tonic tones (specifically, 12 semitones of equal temperament) and key names (major and minor keys). presume. Note that the number of types of keys is not limited to 24 types.

The acoustic analysis device 100 includes a control device 11, a storage device 12, and a display device 13. For example, an information terminal such as a mobile phone, a smartphone, or a personal computer is suitably used as the acoustic analysis device 100. The display device 13 displays the key estimated from the acoustic signal V. The display device 13 is an example of a reproducing device that reproduces the results of analyzing the acoustic signal V. For example, a sound emitting device that emits sound according to the result of analyzing the acoustic signal V may be used as the reproducing device.

The control device 11 is configured with one or more processing circuits such as a CPU (Central Processing Unit), and controls each element of the acoustic analysis device 100. The storage device 12 is one or more memories configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, and stores programs executed by the control device 11 and various data used by the control device 11. Remember. For example, the storage device 12 stores the acoustic signal V. Note that the storage device 12 may be configured by a combination of multiple types of recording media. Alternatively, a portable recording medium that is detachable from the acoustic analysis device 100 or an external recording medium (for example, online storage) with which the acoustic analysis device 100 can communicate via a communication network may be used as the storage device 12. good.

FIG. 2 is a block diagram illustrating functions realized by the control device 11 executing a program stored in the storage device 12. As shown in FIG. The control device 11 implements a feature extraction section 21, a key estimation model 22, and a learning processing section 23. Note that the functions of the control device 11 may be realized by a plurality of devices configured separately from each other. Part or all of the functions of the control device 11 may be realized by a dedicated electronic circuit.

The feature extraction unit 21 extracts the feature amount Y of the acoustic signal V stored in the storage device 12 . The feature extraction unit 21 of the first embodiment includes a first processing unit 211, a second processing unit 212, and a third processing unit 213.

The first processing unit 211 extracts the feature quantity X of the acoustic signal V from the acoustic signal V. The second processing unit 212 estimates the code O from the feature amount X extracted by the first processing unit 211. The third processing unit 213 extracts the feature Y of the acoustic signal V. The feature amount Y is an index representing an acoustic feature by taking into consideration the temporal change of the acoustic signal V. As an example, the third processing section 213 extracts the feature amount Y from the feature amount X extracted by the first processing section 211 and the code O estimated by the second processing section 212. The time series of the feature Y is input to the key estimation model 22.

The key estimation model 22 is a trained model that has learned the relationship between the time series of the feature quantity Y and the key.
Specifically, the key estimation model 22 generates information representing a key (hereinafter referred to as "key information H") by inputting a time series of feature quantities Y.

FIG. 3 is an explanatory diagram of the feature amount X, the feature amount Y, and the key information H. The feature quantity X is extracted every unit period T (T1, T2, T3,...). The unit period T is, for example, a period corresponding to one beat of a song. That is, a time series of feature amounts X is generated from the acoustic signal V. Note that a unit period T of fixed length or variable length may be defined regardless of the beat of the song.

The feature amount X is an index representing the acoustic feature of the portion of the acoustic signal V that corresponds to each unit period T. Code O is estimated for each feature amount X (that is, for each unit period T). That is, a time series of code O is generated. For example, among the plurality of feature quantities X associated with different codes, the code associated with the feature quantity X most similar to the feature quantity X extracted by the first processing unit 211 is estimated as the code O. Note that a statistical estimation model (for example, a hidden Markov model or a neural network) that generates the code O based on the input of the acoustic signal V may be used for estimating the code O.

The feature quantity Y is extracted for each of a series of sections (hereinafter referred to as "continuous sections") U in which the same code O estimated by the second processing unit 212 continues. The interval in which the same code is estimated by the second processing unit 212 is the continuous interval U. A plurality of continuous sections U (U1, U2, U3...) are estimated within the song. For example, one feature value Y is extracted for the continuous section U1 (section corresponding to the unit period T1-T4) in which "F" is estimated as the code O.

FIG. 4 is a diagram schematically showing the feature amount X and the feature amount Y. The feature quantity X includes a chroma vector (PCP: Pitch Class Profile) including a plurality of elements each corresponding to a plurality of scale tones (specifically, 12 semitones of equal temperament), and the intensity Pv of the acoustic signal V. A scale note is a note name (pitch class) that ignores differences in octaves. An element of the chroma vector that corresponds to an arbitrary scale note is set to an intensity (hereinafter referred to as "component intensity") Pq that is the sum of the intensities of the components of the acoustic signal V that correspond to the scale note in question over a plurality of octaves. The feature amount X includes a chroma vector and an intensity Pv for each of a band on the lower side and a band on the higher side than the predetermined frequency. In other words, the chroma vector related to the lower band of the acoustic signal V, the intensity Pv of the acoustic signal V within the band, the chroma vector related to the higher band of the acoustic signal V, and the acoustic signal within the band. The feature amount X includes the intensity Pv of V. That is, the feature quantity X is expressed as a 26-dimensional vector as a whole.

The feature value Y is the variance σq and average μq regarding the time series of the component strength Pq of each scale note, and the variance σv and average μv regarding the time series of the intensity Pv of the acoustic signal V. including for each of the bands. The third processing unit 213 calculates the variance σq and average μq of the component strength Pq included in each of the plurality of feature quantities X within the continuous interval U (that is, the time series of the component strength Pq within the continuous interval U), and the The feature quantity Y is extracted by calculating the variance σv and average μv of the intensity Pv (that is, the time series of the intensity Pv within the continuous interval U) included in each of the plurality of feature quantities X in the continuous interval U. The feature amount Y is expressed as a 52-dimensional vector as a whole. As can be understood from the above explanation, the feature value Y of each continuous section U is an index (typically a variance σq, etc.) related to the temporal change of the continuous section U in the component strength Pq corresponding to the scale note in the acoustic signal V. dispersion).

FIG. 5 is an explanatory diagram of key information H. The key information H includes an index Q (Q1 to Q24) indicating whether each of the 24 types of keys that are candidates for estimation corresponds to the key of the continuous section U. For example, the index Q corresponding to an arbitrary key expresses whether or not the key corresponds to the key of the continuous section U in a binary manner. That is, the key information H is information indicating any one of a plurality of keys. By inputting the feature amount Y into the key estimation model 22 for each continuous section U, key information H is generated for each continuous section U. That is, a time series of key information H is generated by the key estimation model 22. As understood from the above description, the key estimation model 22 is a statistical estimation model that estimates the key of each continuous section U from the time series of the feature Y. In other words, the time series of keys in the song is estimated.

The key estimation model 22 includes a program (for example, a program module that constitutes artificial intelligence software) that causes the control device 11 to execute an operation for generating key information H from a time series of feature values Y, and a plurality of coefficients applied to the operation. This is realized in combination with K. The plurality of coefficients K are set by machine learning (particularly deep learning) using a plurality of teacher data and are stored in the storage device 12. For example, a recurrent neural network (RNN) such as a long short term memory (LSTM) suitable for processing time series data is used as the key estimation model 22.

FIG. 6 is a flowchart illustrating a specific procedure for estimating the key information H of each continuous section U from the acoustic signal V (hereinafter referred to as "key estimation process"). For example, the key estimation process is started in response to an instruction from the user. When the key estimation process is started, the feature extraction unit 21 extracts the feature amount Y for each continuous section U from the acoustic signal V stored in the storage device 12 (Sa1). The key estimation model 22 generates key information H from the time series of the feature amount Y extracted by the feature extraction unit 21 (Sa2). The control device 11 causes the display device 13 to display the key represented by the key information H output from the key estimation model 22 for each continuous section U (Sa3). Note that the content of the display screen showing the result of estimation by the key estimation model 22 is arbitrary. For example, a display screen is displayed that shows both the time series of chords estimated by the second processing unit 212 and the key estimated by the key estimation model 22. Note that, depending on the key estimated by the key estimation model 22, the notations of chords having common constituent notes may be displayed. For example, the code "G♭" is displayed for a continuous section U in which the key "D♭major" is estimated, and the code "F#" is displayed for a continuous section U in which the key "Bmajor" is estimated.

The learning processing unit 23 sets a plurality of coefficients K of the key estimation model 22 by machine learning (particularly deep learning). The learning processing unit 23 sets a plurality of coefficients K by machine learning using a plurality of teacher data L. FIG. 7 is an explanatory diagram of the operation of the learning processing section 23. Each of the plurality of teacher data L is composed of a combination of a time series of feature quantity Y and a time series of key information Hx. The time series of tone information Hx of each teacher data L corresponds to the correct value for the time series of the feature Y in the teacher data L. The teacher data L includes a time series of feature values Y extracted from the performance sounds of an existing song and a time series of key information H representing the key of the song.

The learning processing unit 23 inputs the time series of the feature Y of the teacher data L, and calculates the time series of the key information H output from the provisional key estimation model 22 and the key information Hx of the teacher data L. A plurality of coefficients K of the key estimation model 22 are updated so that the difference is reduced. Specifically, the learning processing unit 23 repeatedly updates the plurality of coefficients K by, for example, error backpropagation so that the evaluation function representing the difference between the key information H and the key information Hx is minimized. A plurality of coefficients K set by the learning processing section 23 through the above procedure are stored in the storage device 12. Therefore, the key estimation model 22 statistically calculates the time series of the unknown feature value Y based on the latent tendency between the time series of the feature value Y in the plurality of teaching data L and the key information Hx. Output valid key information H.

As explained above, the key information H is generated by inputting the feature Y of the acoustic signal V to the key estimation model 22 that has learned the relationship between the feature Y of the acoustic signal V and the key. The key can be estimated with high precision compared to a configuration in which key information H of a song is generated according to the following.

<Second embodiment>
A second embodiment of the present invention will be described. In each of the following examples, for elements whose functions are similar to those in the first embodiment, the reference numerals used in the description of the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

There is a tendency that the key of a song is unlikely to change in a short period of time. In the second embodiment, the key estimated in the first embodiment is corrected based on the above tendency.

FIG. 8 is a block diagram illustrating the functional configuration of the acoustic analysis device 100 according to the second embodiment. The acoustic analysis device 100 of the second embodiment has a configuration in which a post-processing section 24 is added to the acoustic analysis device 100 of the first embodiment. The feature extraction unit 21, key estimation model 22, and learning processing unit 23 are the same as those in the first embodiment. FIG. 9 schematically shows the key time series Wa estimated by the key estimation model 22. The key interval I (I1, I2, I3, . . . ) in FIG. 9 is an interval in which keys represented by the key information H generated by the key estimation model 22 are continuous. As illustrated in FIG. 9, one key interval I includes one or more consecutive continuous intervals U in which the same key is estimated.

The post-processing unit 24 modifies the key time series Wa estimated by the key estimation model 22. Specifically, the post-processing unit 24 generates the time series Wb by correcting the key of the key interval I whose time length is less than a predetermined value among the plurality of key intervals I corresponding to the key time series Wa. . The predetermined value is, for example, a time length corresponding to three beats of a song. In FIG. 9, a case is illustrated in which the key interval I2 in which the key "Emajor" is estimated is less than a predetermined value. The post-processing unit 24 of the second embodiment replaces the key of the key section I with the key represented by the key information H immediately before the key section I when the time length of the key section I is less than a predetermined value. Specifically, the key "Emajor" of the key interval I2 is replaced with the key "Fmajor" represented by the key information H of the continuation interval U immediately before the key interval I2 (that is, the continuation interval U at the end of the key interval I1). .

FIG. 10 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as "post-processing 1") for correcting the key estimated by the key estimation model. After the key time series Wa is estimated by the key estimation model, post-processing 1 is started, for example, in response to an instruction from the user. When the post-processing 1 is started, the post-processing unit 24 defines a plurality of key intervals I (I1, I2, I3, . . . ) on the time axis from the key time series Wa (Sb1). That is, the time series of key interval I is specified. The post-processing unit 24 selects one of the plurality of key intervals I (Sb2). Specifically, key intervals I are sequentially selected from the beginning to the end. The post-processing unit 24 determines whether the time length of the key interval I selected in step Sb2 is less than a predetermined value (Sb3). If the time length of the key interval I is less than the predetermined value (Sb3: YES), the post-processing unit 24 replaces the key of the key interval I with the key represented by the key information H immediately before the key interval I (Sb4). . On the other hand, if the time length of the key section I exceeds the predetermined value (Sb3: NO), the key is not corrected, and the key section I located immediately after the key section I is selected (Sb2). When determining the time length (Sb3) for each of all key sections I and correcting the key of the key section I that is shorter than a predetermined value (Sb5: YES), the control device 11 controls the post-processing section 24 The key time series Wb generated by is displayed on the display device 13 (Sb6). That is, the display device 13 displays the time series Wb in which the key has been replaced for all the key intervals I whose time lengths are less than a predetermined value among the plurality of key intervals I. On the other hand, if there is an unselected key interval I in step Sb2 (Sb5: NO), the post-processing unit 24 repeatedly executes the processes of steps Sb2 to Sb4 for the unselected key interval I. Note that when the key in the first key section I in the time series of key sections I is to be replaced, it is replaced with the key represented by the key information H immediately after the key section I.

The second embodiment also achieves the same effects as the first embodiment. In the second embodiment, when the time length of a key interval I in which keys represented by key information H are consecutive is less than a predetermined value, the key of the relevant key interval I is changed to the key represented by the key information H immediately before the relevant key interval I. will be replaced with Therefore, the key estimated by the key estimation model can be appropriately corrected, taking into account the tendency that the key is unlikely to change in a short period of time. In addition, in the second embodiment, the key represented by the key information H of the key interval I immediately before the replacement source is set as the replacement destination, but the key represented by the key information H of the key interval I immediately after the replacement source is also used as the replacement destination. good. In the above configuration, for example, in step Sb2 of FIG. 10, the key sections I are sequentially selected from the end to the beginning among the plurality of key sections I.

<Third embodiment>
There is a correlation between the chord and key of a song. For example, a chord whose constituent tones correspond to the scale of the key of the song is played within the song. In particular, there is a tendency that the first chord of a section in which a specific key is set in a song is likely to match a chord whose root note is the tonic of the key. In the third embodiment, the key estimated in the first embodiment is corrected based on the above tendency.

FIG. 11 is a block diagram illustrating the functional configuration of an acoustic analysis device 100 according to the third embodiment. The acoustic analysis device 100 of the third embodiment includes a post-processing section 24 similarly to the second embodiment. The post-processing unit 24 generates a time series Wb by modifying the key time series Wa estimated by the key estimation model 22. The post-processing unit 24 of the third embodiment generates the time series Wb using the time series of the code O in the acoustic signal V (for example, the time series of the code O estimated by the second processing unit 212). FIG. 12 is an explanatory diagram of the time series Wb generated by the post-processing unit 24 according to the third embodiment. Specifically, the post-processing unit 24 changes the end point (specifically, the starting point S) of the key interval I according to the time series of the code O in the acoustic signal V. The post-processing unit 24 of the third embodiment adds the key information H of the key interval I to the time series of the code O of the acoustic signal V in the section R (hereinafter referred to as "search section") including the starting point S of the key interval I. If there is a chord corresponding to the key represented (hereinafter referred to as "key corresponding chord"), change the start point S of the key section I to the start point of the section (typically the continuation section U) corresponding to the key corresponding chord. do. The search section R is exemplified by, for example, a plurality of (six in FIG. 12) continuous sections U centered on the starting point S of the key section I. A key-compatible chord is, for example, a chord whose root note is the tonic of the key (typically a tonic chord). In FIG. 12, the starting point S of the key interval I2 in which "Emajor" is estimated is changed to the starting point of the continuation interval U in which the code "E" is estimated among the plurality of codes O estimated for the search interval R. is exemplified.

FIG. 13 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as "post-processing 2") in which the post-processing unit 24 according to the third embodiment corrects the key estimated by the key estimation model. After the key time series Wa is estimated by the key estimation model 22, the post-processing unit 24 starts post-processing 2, for example, in response to an instruction from the user. When post-processing 2 is started, the post-processing unit 24 defines a plurality of key intervals I on the time axis from the key time series Wa (Sc1). That is, the time series of key interval I is specified. The post-processing unit 24 selects one of the plurality of tonal intervals I (Sc2). Specifically, key intervals I are sequentially selected from the beginning to the end.

Next, the post-processing unit 24 compares each of the plurality of chords (hereinafter referred to as "candidate chords") estimated for the search interval R in the key interval I selected in step Sc2 with the key corresponding code. The degree of similarity between the two is calculated (Sc3). The degree of similarity is an index representing the distance or correlation between a vector representing a key-corresponding chord and a vector representing a candidate chord. Vectors representing key-corresponding chords or candidate chords include, for example, the literature ``Naohiko Yamaguchi, Noboru Kanmura, ``Expansion of TPS (Tonal Pitch Space) to accommodate chords including non-tonal constituent notes - Towards jazz music theory.'' The basic space function described in "Aiming for the application of ``Information Processing Society of Japan Research Report, February 11, 2011'' is suitable. Next, the post-processing unit 24 searches for a candidate code having the maximum degree of similarity with the key-corresponding code from among the plurality of candidate codes (Sc4). Step Sc3 and Step Sc4 search for one candidate chord that is most similar (typically matches) to the key-corresponding chord represented by the key of key interval I from a plurality of candidate chords in search interval R in key interval I. This is the process of

The post-processing unit 24 changes the starting point S of the key interval I to the starting point of the interval corresponding to the candidate chord searched in step Sc4 (Sc5). Specifically, the starting point S of the key interval I is changed to the starting point of the continuation interval U from which the candidate chord has been estimated. Note that when the similarity of the candidate chords at the beginning of the key interval I is the maximum, the starting point S of the key interval I is maintained unchanged. When the processes of steps Sc2 to Sc6 are executed for each of all the key intervals I (Sc6: YES), the control device 11 causes the display device 13 to display the time series Wb generated by the post-processing section 24 (Sc7). That is, the display device 13 displays the time series Wb in which the starting point S is changed or maintained for each of the plurality of adjustment intervals I defined from the time series Wa. On the other hand, if there is an unselected key interval I in step Sc2 (Sc6: NO), the post-processing unit 24 executes the processes of steps Sc2 to Sc6 for the unselected key interval I.

The third embodiment also achieves the same effects as the first embodiment. In the third embodiment, since the end points of the key interval I are changed according to the time series of the chord O in the acoustic signal V, the key estimated by the key estimation model 22 is appropriately adjusted by taking into account the time change of the chord. can be corrected. In addition, in the third embodiment, the time series of chords (that is, a plurality of candidate chords) of the acoustic signal V in the search interval R including the starting point S of the key interval I includes the tonic of the key represented by the key information H of the key interval I. When there is a key-corresponding chord having the root note , the starting point S of the key interval I is changed to the starting point of the section corresponding to the key-corresponding chord. Therefore, taking into consideration the tendency that the beginning of the time series of the chord O of the acoustic signal V corresponding to the key interval I is likely to be a chord whose root note is the tonic of the key, the key information H is appropriately adjusted. Can be fixed. Note that the post-processing 1 in the second embodiment and the post-processing 2 in the third embodiment may be combined.

<Modified example>
Specific modification modes added to each of the embodiments exemplified above are illustrated below. A plurality of aspects arbitrarily selected from the examples below may be combined as appropriate to the extent that they do not contradict each other.

(1) In each of the above embodiments, the acoustic analysis device 100 that includes the key estimation model 22 and the learning processing section 23 is illustrated, but an information processing device (hereinafter referred to as a “machine learning device”) separate from the acoustic analysis device 100 is illustrated. ) may be equipped with the learning processing unit 23. The key estimation model 22 to which the plurality of coefficients K set by the learning processing unit 23 of the machine learning device is applied is transferred to the acoustic analysis device 100 and used for generating key information H. As understood from the above description, the learning processing section 23 is omitted from the acoustic analysis device 100.

(2) The acoustic analysis device 100 may be realized by a server device that communicates with an information terminal such as a mobile phone or a smartphone. For example, the acoustic analysis device 100 generates key information H by analyzing the acoustic signal V received from the information terminal and transmits it to the information terminal. Note that in a configuration in which the feature extraction unit 21 that extracts the feature Y from the acoustic signal V is installed in the information terminal, the acoustic analysis device 100 inputs the time series of the feature Y received from the information terminal to the key estimation model 22. By doing so, tone information H is generated and the tone information H is transmitted to the information terminal. As understood from the above description, the feature extraction unit 21 may be omitted from the acoustic analysis device 100.

(3) In each of the above-mentioned embodiments, for example, the time series of known chords O specified in the musical score of a song may be used to extract the feature Y. The third processing unit 213 generates the feature quantity Y from the feature quantity X extracted by the first processing unit 211 and the time series of the known code O. That is, the second processing unit 212 may be omitted. In the above configuration, the display device 13 displays a display screen on which the time series of the known chord O and the time series of the key estimated by the key estimation model 22 are written together. Note that the time series of the known code O is stored in the storage device 12 in advance.

(4) In each of the above embodiments, the key information H is generated for each continuous section U by inputting the feature value Y for each continuous section U into the key estimation model 22. However, the input and output to the key estimation model 22 is not limited to the above examples. For example, the following configurations [A] to [D] are adopted.

[A] By inputting the time series of the feature amount X for each unit period T generated by the first processing unit 211 to the key estimation model 22, key information H is generated for each unit period T. That is, the key estimation model 22 learns the relationship between the time series of the feature amount X and the key information H. Note that the second processing section 212 and the third processing section 213 are omitted.

[B] Input the time series of feature amount X for each unit period T generated by the first processing unit 211 and the time series of code O for each unit period T generated by the second processing unit 212 to the key estimation model 22 By doing so, key information H is generated for each unit period T. That is, the key estimation model 22 learns the relationship between the time series of the feature amount X, the time series of the chord O, and the key information H. Note that the third processing unit 213 is omitted.

[C] The key information H may be generated for each unit period T by inputting the time series of the chord O for each unit period T generated by the second processing unit 212 to the key estimation model 22. That is, the key estimation model 22 learns the relationship between the time series of the chord O and the key information H. The third processing unit 213 is omitted. Note that in [B] and [C], for example, a time series of chord O generated from a known musical score of a song may be used as an input to the key estimation model 22.

[D] Input the time series of the feature quantity X for each unit period T generated by the first processing unit 211 and the data representing the time series of the continuous interval U (hereinafter referred to as "interval data") to the key estimation model 22. Thus, key information H is generated for each continuous section U. That is, the key estimation model 22 learns the relationship between the time series and interval data of the feature amount X and the key information H. Note that the third processing unit 213 is omitted. The section data is, for example, data indicating the boundary of the continuous section U, and may be generated, for example, from the time series of the chord O generated by the second processing unit 212, or may be, for example, section data generated from the known musical score of the song. You may also use

As understood from the above description, inputs and outputs to the key estimation model 22 are arbitrary. Note that the units of input and output can be changed as appropriate depending on the type of input. For example, input and output for each continuous section U or for each unit period T are exemplified. Furthermore, the configuration of the feature extraction unit 21 can be changed as appropriate depending on the input to the key estimation model 22.

(5) In each of the above-described embodiments, the key information H including the index Q that binaryly represents one of a plurality of keys is illustrated, but the content of the key information H is not limited to the above examples. For example, the index Q corresponding to each key may use key information H indicating the likelihood that the key of the song corresponds to that key. The index Q representing the likelihood is set to a numerical value within the range of 0 or more and 1 or less, and the total value of the index Q over all keys corresponding to different keys is a predetermined value (for example, 1). Further, the key estimation model 22 may generate the key information H as identification information for identifying one of a plurality of keys.

(6) In each of the above-described embodiments, the feature amount X including the chroma vector and the intensity Pv is illustrated, but the content of the feature amount X is not limited to the above examples. For example, the chroma vector may be used alone as the feature Y. In addition, although the feature quantity X including the chroma vector and the intensity Pv for each of the low-frequency band component and the high-frequency band component with respect to a predetermined frequency of the acoustic signal V is illustrated, A feature amount X including a chroma vector and intensity Pv may be generated for the frequency band. Similarly, the feature value Y is the variance σq and average μq of the time series of the component strength Pq of each scale note, and the variance σv and average μv of the time series of the intensity Pv of the acoustic signal V. It may be included for the entire frequency band.

(7) In the post-processing 2 of the third embodiment, structural sections in the musical structure of the song corresponding to the acoustic signal V (for example, phrases such as the A melody, chorus, B melody, etc.) may be taken into account. For example, there is a tendency for the key to change from one structural section to another. Utilizing the above tendency, for example, if the starting point of the structural section is located within the search section R of the key section I, the starting point S of the key section I may be changed to the starting point of the structural section.

(8) In each of the above embodiments, the key represented by the key information H generated by the key estimation model 22 is displayed, but the use of the key information H is not limited to the above examples. If the chord displayed by the display device 13 (the chord estimated by the second processing unit 212) is, for example, a chord that is difficult for the user to play, there is a situation in which the user wants to change the chord to a simpler chord. In consideration of the above circumstances, the key estimated by the key estimation model 22 may be used to identify a plurality of chords that are candidates for change by the user. Taking into account the key estimated by the key estimation model 22, a plurality of acoustically similar chords are identified as candidates for change.

(9) In the above embodiment, the key estimation model estimates the key in equal temperament, but the temperament that is the basis of the key estimated by the key estimation model is not limited to equal temperament. For example, the key estimation model may estimate the key in the temperament of ethnic music such as Indian music.

(10) The acoustic analysis device 100 according to each of the above embodiments is realized by cooperation between a computer (specifically, the control device 11) and a program, as illustrated in each embodiment. The programs according to each of the above embodiments may be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but any known recording medium such as a semiconductor recording medium or a magnetic recording medium is used. may include a recording medium in the form of. Note that the non-transitory recording medium includes any recording medium excluding transitory, propagating signals, and does not exclude volatile recording media. It is also possible to provide the program to a computer in the form of distribution via a communication network.

Note that the main body that executes the program that implements the key estimation model 22 is not limited to a general-purpose processing circuit such as a CPU. For example, a processing circuit specialized for artificial intelligence such as a Tensor Processing Unit or a Neural Engine, or an electronic circuit for signal processing (DSP: Digital Signal Processor) may execute the program. Furthermore, multiple types of entities selected from the above examples may cooperate to execute the program.

(11) The trained model (key estimation model 22) is a statistical estimation model (for example, a neural network) realized by a control device (eg, a computer), and generates an output B according to an input A. Specifically, the learned model includes a program (for example, a program module that configures artificial intelligence software) that causes a control device to execute a calculation that specifies output B from input A, and a plurality of coefficients that are applied to the calculation. Realized by combination. The plurality of coefficients of the trained model are optimized by prior machine learning (deep learning) using a plurality of teacher data in which input A and output B are made to correspond. That is, the trained model is a statistical estimation model that has learned the relationship between input A and output B. The control device performs an operation applying a plurality of learned coefficients and a predetermined response function to an unknown input A, thereby detecting a tendency latent in a plurality of training data (between input A and output B). A statistically valid output B is generated for the input A under the following relationship.

(12) From the embodiments exemplified above, the following configurations can be understood, for example.

The acoustic analysis method according to the aspect (first aspect) of the present invention involves inputting the time series of the features of the acoustic signal to a trained model that has learned the relationship between the time series of the features of the acoustic signal and the key. , generate key information representing the key. According to the aspect described above, key information representing the key is generated by inputting the feature amount of the audio signal to the trained model that has learned the relationship between the feature amount of the audio signal and the key, so that the key information representing the key is generated. The key can be estimated with high accuracy compared to a configuration that generates key information.

In an example of the first aspect (second aspect), the feature amount of the acoustic signal is input to the learned model for each continuous section in which the same chord continues, thereby generating the key information for each continuous section. According to the above aspect, by inputting the feature amount of the acoustic signal into the learned model for each continuous section in which the same chord continues, key information is generated for each continuous section, so the key information is generated for each continuous section in which the same chord continues. Taking into consideration the tendency that the key does not change within the interval, key information can be estimated with high accuracy.

In an example of the second aspect (third aspect), the feature amount of each continuous section includes, for each scale note, an index regarding a temporal change in the continuous section in the component intensity corresponding to the scale note of the acoustic signal. According to the above aspect, since the feature amount including the index regarding the time change of the continuous interval in the component intensity corresponding to the scale note in the acoustic signal is input for each continuous interval to the trained model for each continuous interval, the acoustic signal It is possible to estimate the key information with high accuracy by taking into account the temporal change in the .

In an example of the first aspect (fourth aspect), the feature amount includes, for each scale note, component intensities obtained by adding component intensities corresponding to scale notes in the acoustic signal over a plurality of octaves. According to the above aspect, since the feature amount including the component strength obtained by adding the component intensities corresponding to the scale notes of the acoustic signal over multiple octaves is input to the trained model for each scale note, the music represented by the acoustic signal is input into the trained model. This has the advantage that key information can be estimated with high precision by using features that appropriately reflect the chord.

In an example of any one of the first to fourth aspects (fifth aspect), when the time length of a key interval in which the key represented by the key information is continuous is less than a predetermined value, the key of the key interval is Replace with the key indicated by the key information immediately before or after the section. According to the above aspect, when the time length of a key interval in which the keys represented by the key information are continuous is less than a predetermined value, the key of the relevant key interval is replaced with the key represented by the key information immediately before or after the relevant key interval. be done. Therefore, the key estimated by the learned model can be appropriately corrected, taking into account the tendency that the key is unlikely to change in a short period of time.

In an example of any one of the first to fifth aspects (sixth aspect), the end points of a key interval in which keys represented by the key information are continuous are changed in accordance with the time series of chords in the acoustic signal. According to the above aspect, since the end point of the key interval in which the keys represented by the key information are consecutive is changed according to the time series of the chords in the acoustic signal, it is estimated by the trained model, taking into account the time change of the chord. It is possible to appropriately correct the key that has been created.

In an example of the sixth aspect (seventh aspect), the time series of chords of the acoustic signal in the search section including the start point of the key section includes a code whose root note is the tonic of the key represented by the key information of the key section. In this case, the start point of the key interval is changed to the start point of the interval corresponding to the chord. According to the above aspect, in the time series of the chords of the acoustic signal in the search section including the start point of the key section, a code ( Ideally, if there is a matching chord), the start point of the key section is changed to the start point of the section corresponding to the chord. Therefore, it is highly likely that the beginning of the time series of chords in the acoustic signal corresponding to a key interval is a chord that acoustically approximates (ideally, a matching chord) a chord whose root is the tonic of the key. By taking this tendency into consideration, the key information can be appropriately corrected.

An acoustic analysis device according to a preferred aspect (eighth aspect) of the present invention is a trained model that has learned the relationship between the time series of feature quantities of an acoustic signal and the key, It includes a key estimation model that generates key information representing a key from an input. According to the aspect described above, key information representing the key is generated by inputting the feature amount of the audio signal to the trained model that has learned the relationship between the feature amount of the audio signal and the key, so that the key information representing the key is generated. The key can be estimated with high accuracy compared to a configuration that generates key information.

DESCRIPTION OF SYMBOLS 100... Acoustic analysis device, 11... Control device, 12... Storage device, 13... Display device, 21... Feature extraction part, 211... Processing part, 212... Processing part, 213... Processing part, 22... Key estimation model, 23... Learning processing unit, 24... Post-processing unit.

Claims (8)

a first processing unit that extracts a first feature from the acoustic signal;
a third processing unit that generates a second feature amount from the first feature amount and a code corresponding to the acoustic signal; and
A trained model that has learned a relationship between a second feature amount and a key, and is a key estimation model that generates key information representing a key from a time series input of the second feature amount generated by the third processing unit. ,
A program that makes a computer function as moreover,
causing the computer to function as a second processing unit that estimates the code from the acoustic signal;
The program according to claim 1 , wherein the third processing section generates the second feature amount from the first feature amount extracted by the first processing section and the code estimated by the second processing section. By inputting the time series of the features of the acoustic signal for each continuous section in which the same chord continues, to the trained model that has learned the relationship between the feature and the key, key information representing the key can be calculated for each continuous section. A computer-implemented acoustic analysis method. An acoustic analysis method realized by a computer that generates key information representing a key by inputting a time series of features of an acoustic signal to a trained model that has learned the relationship between features and keys,
The feature quantity includes, for each scale note, an index related to a temporal change in the intensity of a component corresponding to a scale note in the acoustic signal. By inputting the time series of the features of the acoustic signal to a trained model that has learned the relationship between the features and the key, key information representing the key is generated.
When the time length of a key interval in which the key represented by the key information is continuous is less than a predetermined value, the key of the concerned key interval is replaced with the key represented by the key information immediately before or after the concerned key interval.Implemented by a computer. Acoustic analysis method. By inputting the time series of the features of the acoustic signal to a trained model that has learned the relationship between the features and the key, key information representing the key is generated.
An acoustic analysis method realized by a computer, wherein an end point of a key interval in which keys represented by the key information are continuous is changed according to a time series of chords in the acoustic signal, the method comprising:
In changing the end point of the key interval, if there is a key-corresponding code corresponding to the key represented by the key information of the key interval in the time series of codes of the acoustic signal in the search interval including the start point of the key interval, the end point of the key interval is changed. An acoustic analysis method that changes the start point of a key interval to the start point of an interval corresponding to the key corresponding chord. 7. The acoustic analysis method according to claim 6 , wherein the key corresponding chord is a chord whose root note is the tonic of the key represented by the key information of the key interval. By inputting the time series of the features of the acoustic signal for each continuous section in which the same chord continues, to the trained model that has learned the relationship between the feature and the key, key information representing the key can be calculated for each continuous section. An acoustic analysis device that generates

Images