JP7243147B2 - Code estimation method, code estimation device and program - Google Patents

Description

The present invention relates to a technique for discriminating chords from acoustic signals representing sounds such as voices or musical tones.

Conventionally, there has been proposed a technique for specifying a chord name from an acoustic signal representing sound such as singing sound or performance sound of a piece of music. For example, Japanese Laid-Open Patent Publication No. 2002-100000 discloses a technique for judging chords based on a frequency spectrum analyzed from waveform data of input musical tones. Chords are identified by matching information on the frequency spectrum with chord patterns prepared in advance. Further, Patent Literature 2 discloses a technique for identifying a chord including constituent tones of the fundamental frequency for which a peak is observed in the probability density function of the fundamental frequency of the input sound. Patent Literature 3 discloses a technique of estimating a code using a machine-learned neural network.

JP-A-2000-298475 JP 2008-209550 A JP 2017-215520 A

However, with the technique of Patent Literature 1, it is not possible to estimate an appropriate chord pattern with high accuracy when the information on the analyzed frequency spectrum is extremely deviated from the chord pattern prepared in advance. An object of the present invention is to estimate a code with high accuracy.

In order to solve the above problems, a chord estimation method according to a preferred aspect of the present invention estimates a first chord from an acoustic signal and inputs the first chord to a trained model that has learned the tendency of changes to chords. to estimate the second code.

1 is a block diagram showing the configuration of a code estimation device according to a first embodiment of the present invention; FIG. 2 is a block diagram showing a functional configuration of a code estimation device; FIG. FIG. 4 is a schematic diagram of data generated until a second code is estimated from an acoustic signal; It is a schematic diagram of a 1st feature-value and a 2nd feature-value. 2 is a block diagram showing a functional configuration of a machine learning device; FIG. 4 is a flowchart of code estimation processing; 10 is a flowchart of processing for estimating a second code; FIG. 8 is a block diagram of a code estimator according to the second embodiment; FIG. 11 is a block diagram of a code estimation unit according to the third embodiment; FIG. FIG. 12 is a block diagram of a code estimation unit according to the fourth embodiment; FIG. FIG. 12 is a block diagram showing a functional configuration of a chord estimation device according to a fifth embodiment; FIG. FIG. 4 is an explanatory diagram of boundary data; FIG. 14 is a flowchart of chord estimation processing in the fifth embodiment; FIG. FIG. 21 is an explanatory diagram of machine learning of a boundary estimation model in the fifth embodiment; FIG. 12 is a block diagram showing a functional configuration of a chord estimation device according to a sixth embodiment; FIG. 16 is a flowchart of processing for estimating a second code in the sixth embodiment; FIG. FIG. 20 is an explanatory diagram of machine learning of a chord transition model in the sixth embodiment;

<First Embodiment>
FIG. 1 is a block diagram illustrating the configuration of a code estimation device 100 according to the first embodiment of the present invention. A chord estimation device 100 of the first embodiment is a computer system that estimates chords from an acoustic signal V representing performance sounds of a piece of music (for example, singing voices or musical tones). In the first embodiment, a server device that estimates the code time series from the acoustic signal V transmitted by the terminal device 300 and transmits the estimated code time series to the terminal device 300 is used as the code estimation device 100 . The terminal device 300 is, for example, a portable information terminal such as a mobile phone or a smart phone, or a portable or stationary information terminal such as a personal computer. It can communicate with the device 100 .

Specifically, the code estimation device 100 includes a communication device 11 , a control device 12 and a storage device 13 . The communication device 11 is a communication device that communicates with the terminal device 300 via a communication network. Communication by the communication device 11 may be either wired communication or wireless communication. The communication device 11 of the first embodiment receives the acoustic signal V transmitted from the terminal device 300 . The control device 12 is, for example, a processing circuit such as a CPU (Central Processing Unit), and comprehensively controls each element constituting the code estimation device 100 . The controller 12 comprises at least one circuit. The control device 12 of the first embodiment estimates the code time series from the acoustic signal V transmitted from the terminal device 300 .

The storage device (memory) 13 is composed of a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of a plurality of types of recording media. Stores various data. Note that a storage device 13 (for example, cloud storage) is prepared separately from the code estimation device 100, and the control device 12 executes writing and reading to and from the storage device 13 via a communication network such as a mobile communication network or the Internet. You may That is, the storage device 13 can be omitted from the code estimation device 100 .

FIG. 2 is a block diagram illustrating the functional configuration of the control device 12. As shown in FIG. The control device 12 performs a plurality of tasks according to a program stored in the storage device 13, thereby providing a plurality of functions (first extraction unit 21, analysis unit 23, second extraction 25 and code estimator 27). Note that the functions of the control device 12 may be implemented by a set of multiple devices (that is, a system), or some or all of the functions of the control device 12 may be implemented by a dedicated electronic circuit (for example, a signal processing circuit). good too.

The first extraction unit 21 extracts the first feature Y1 of the acoustic signal V from the acoustic signal V. As shown in FIG. The first feature Y1 is extracted for each unit period T (T1, T2, T3, . . . ) as illustrated in FIG. The unit period T is, for example, a period corresponding to one beat of music. That is, a time series of the first feature Y1 is generated from the acoustic signal V. FIG. It should be noted that the fixed-length or variable-length unit period T may be defined regardless of the beat of the music.

The first feature amount Y1 is an index representing the acoustic features of the portion of the acoustic signal V corresponding to each unit period T. As shown in FIG. FIG. 4 schematically shows the first feature Y1. As an example, the first feature amount Y1 includes a chroma vector (PCP: Pitch Class Profile) including a plurality of elements corresponding to a plurality of scale tones (e.g., 12 semitones of equal temperament) and the strength Pv of the acoustic signal V. . Scale notes are note names (pitch classes) ignoring octave differences. An element of the chroma vector corresponding to an arbitrary scale tone is set to an intensity (hereinafter referred to as "component intensity") Pq obtained by adding the intensities of the components of the acoustic signal V corresponding to the scale tone in question over a plurality of octaves. The first feature amount Y1 of the first embodiment includes the chroma vector and the intensity Pv for each of the band on the lower side and the band on the higher side than the predetermined frequency. That is, the chroma vector for the low-frequency band of the audio signal V, the intensity Pv of the audio signal V in the band, the chroma vector for the high-frequency band of the audio signal V, and the audio signal in the band The intensity Pv of V is included in the first feature Y1. That is, the first feature Y1 is expressed as a 26-dimensional vector as a whole.

The analysis unit 23 in FIG. 2 estimates the first code X1 from the first feature Y1 extracted by the first extraction unit 21. FIG. As illustrated in FIG. 3, the first code X1 is estimated for each first feature Y1 (that is, each unit period T). That is, a time series of the first code X1 is generated. The first code X1 is an initial or provisional code corresponding to the acoustic signal V. FIG. For example, among a plurality of first feature quantities Y1 associated with different codes, the code associated with the first feature quantity Y1 most similar to the first feature quantity Y1 extracted by the first extraction unit 21 is the first code. estimated as 1 code X1. A statistical estimation model (for example, a hidden Markov model or a neural network) that generates the first code X1 from the input of the acoustic signal V may be used to estimate the first code X1. As can be understood from the above description, the first extraction unit 21 and the analysis unit 23 function as the preprocessing unit 20 that estimates the first code X1 from the acoustic signal V. FIG. The preprocessing unit 20 is an example of a "first chord estimation unit".

The second extractor 25 in FIG. 2 extracts the second feature quantity Y2 from the acoustic signal V. As shown in FIG. The second feature amount Y2 is an index representing an acoustic feature in consideration of changes in the acoustic signal V over time. As an example, the second extraction unit 25 extracts the second feature Y2 from the first feature Y1 extracted by the first extraction unit 21 and the first code X1 estimated by the analysis unit 23 . As exemplified in FIG. 3, the second feature Y2 is extracted for each series of intervals in which the same first code X1 is estimated (hereinafter referred to as "continuous intervals"). For example, one second feature Y2 is extracted for a continuous section (a section corresponding to the unit period T1-T4) in which "F" is estimated as the first code X1. FIG. 4 schematically shows the second feature Y2. The second feature amount Y2 of the first embodiment is the variance σq and average μq regarding the time series of the component strength Pq for each tone of the scale, and the variance σv and average μv regarding the time series of the strength Pv of the acoustic signal V. This is included for each of the side band and the high side band. As exemplified in FIG. 4, the second extraction unit 25 of the first embodiment has a component intensity Pq included in each of the plurality of first feature quantities Y1 in the continuous interval (that is, when the component intensity Pq in the continuous interval series) and the variance σv and average μv of the intensity Pv (that is, the time series of the intensity Pv in the continuous interval) contained in each of the plurality of first feature quantities Y1 in the continuous interval. Thus, the second feature Y2 is extracted. The second feature Y2 is expressed as a 52-dimensional vector as a whole. As can be understood from the above description, the second feature quantity Y2 is an index (typically, the degree of dispersion such as variance σq) relating to the time change of the component strength Pq of each scale sound, and the time Indices of change (typically the degree of scatter, such as variance σv).

By the way, it is also possible to transmit the time series of the first code X1 estimated by the preprocessing unit 20 to the terminal device 300. FIG. However, the first code X1 estimated by the preprocessing unit 20 has room for the user U to change. For example, if the first code X1 is erroneously estimated, or if the first code X1 does not suit the user U's preferences, the first code X1 needs to be changed. Considering the above circumstances, the code estimator 27 in FIG. 2 uses the learned model M to estimate the second code X2 from the first code X1 and the second feature Y2. As illustrated in FIG. 3, the time series of the second code X2 corresponding to the first code X1 is estimated. The trained model M is a predictive model that has learned the tendency of changes to the first code X1, and is generated by machine learning using teacher data indicating the results of changes made to the first code X1 by many users. In other words, the second code X2 is a code that is statistically more valid than the first code X1 under the trend of code changes by many users. The chord estimator 27 is an example of a "second chord estimator".

The code estimation unit 27 includes a trained model M and an estimation processing unit 70, as illustrated in FIG. The trained model M of the first embodiment is composed of a first trained model M1 and a second trained model M2. The first trained model M1 is a prediction model that has learned a tendency (hereinafter referred to as a "first tendency") regarding the content of changes made to the first code X1 by many users. The first tendency is the tendency of what kind of code the first code X1 is changed to. On the other hand, the second trained model M2 is a prediction model that has learned a code change tendency (hereinafter referred to as "second tendency") different from the first tendency. Specifically, the second trend is a trend including a trend regarding the presence or absence of changes to the code and a trend regarding the content of changes to the code. For example, the second tendency is whether or not the first code X1 will be changed, and if so, what kind of code it will be changed to. That is, the first trend is included in the second trend.

The first trained model M1 outputs the probability of appearance λ1 for each of a plurality of codes (hereinafter referred to as "candidate codes") that are candidates for the second code X2 in response to the inputs of the first code X1 and the second feature Y2. do. Specifically, the occurrence probability λ1 is output for each of Q candidate chords having different combinations of root note, type (for example, chord type such as major or minor), and base note (Q is a natural number of 2 or more). . The appearance probability λ1 of the candidate code that is highly likely to be changed from the first code X1 under the first trend is a relatively high numerical value. On the other hand, the second trained model M2 outputs the appearance probability λ2 for each of the Q candidate codes for the input of the first code X1 and the second feature Y2. The appearance probability λ2 of the candidate code that is highly likely to be changed from the first code X1 under the second trend is a relatively high value. Note that "no code" may be included as one of the Q candidate codes.

The estimation processing unit 70 estimates the second code X2 based on the result of estimation by the first trained model M1 and the result of estimation by the second trained model M2. In the first embodiment, the second code X2 is estimated based on the occurrence probability λ1 output by the first trained model M1 and the occurrence probability λ2 output by the second trained model M2. Specifically, the estimation processing unit 70 calculates the appearance probability λ0 for each candidate code by integrating the appearance probability λ1 and the appearance probability λ2 for each candidate code, and the appearance probability λ0 among the Q candidate codes is Presume the high candidate code as the second code X2. That is, a candidate code that is statistically valid for the first code X1 under both the first trend and the second trend is output as the second code X2. The occurrence probability λ0 of each candidate code is, for example, the weighted sum of the occurrence probability λ1 and the occurrence probability λ2. The appearance probability λ0 may be calculated by adding the appearance probability λ1 and the appearance probability λ2, or by inputting the appearance probability λ1 and the appearance probability λ2 into a predetermined function. The time series of the second code X2 estimated by the code estimation unit 27 is transmitted to the terminal device 300 of the user U.

The first trained model M1 is, for example, a neural network (typically a deep neural network) and is defined by a plurality of coefficients K1. Similarly, the second trained model M2 is, for example, a neural network (typically a deep neural network), defined by a plurality of coefficients K2. A plurality of coefficients K1 and a plurality of coefficients K2 are set by machine learning using teacher data L indicating the tendency of code changes by many users. FIG. 5 is a block diagram showing the configuration of a machine learning device 200 for setting multiple coefficients K1 and multiple coefficients K2. Machine learning device 200 is implemented by a computer system that includes teacher data generation unit 51 and learning unit 53 . The teacher data generation unit 51 and the learning unit 53 are realized by a control device (not shown) such as a CPU (Central Processing Unit). Note that the code estimation device 100 may be equipped with the machine learning device 200 .

A storage device (not shown) of the machine learning device 200 stores a plurality of changed data Z for generating teacher data L. FIG. The change data Z are collected in advance from many terminal devices. For example, it is assumed that the analysis unit 23 estimates the time series of the first code X1 from the acoustic signal V in the terminal device of the user. The user confirms whether or not to change each of the plurality of first codes X1 estimated by the analysis unit 23, and if there is a change, inputs the changed code. That is, each change data Z is data representing the history of changes to the first code X1 by the user. When the user completes confirmation of the plurality of first codes X1, change data Z is generated and transmitted to the machine learning device 200. FIG. Each change data Z is transmitted to the machine learning device 200 from terminal devices of many users. Note that the machine learning device 200 may generate the change data Z. FIG.

Each change data Z represents whether or not the first code X1 has been changed by the user and the content of the change for each time series of the first code X1 estimated from the acoustic signal V. FIG. Specifically, as exemplified in FIG. 5, any one change data Z is added to each first code X1 estimated in the terminal device, a confirmed code corresponding to the first code X1 and a second It is a data table registered in association with the feature amount Y2. That is, the change data Z is composed of the time series of the first code X1, the time series of the confirmed codes, and the time series of the second feature Y2. The confirmed code is a code that indicates whether or not the first code X1 has been changed and the content of the change. Specifically, when the user changes the first code X1, the changed code is set as the confirmed code, and when the user does not change the first code X1, the first code X1 is set as the confirmed code. Code X1 is set as the verified code. The second feature Y2 corresponding to the first code X1 is registered in the change data Z generated from the first code X1 and the first feature Y1.

The teacher data generator 51 of the machine learning device 200 generates teacher data L from the modified data Z. FIG. The teacher data generation unit 51 of the first embodiment includes a selection unit 512 and a processing unit 514, as illustrated in FIG. The selection unit 512 selects the change data Z suitable for generating the teacher data L from among the plurality of change data Z. FIG. For example, the change data Z, in which the total number of places where the first code X1 is changed is large, can be evaluated as having high reliability as data representing the tendency of the code change by the user. Considering the above tendency, for example, change data Z is selected in which the total number of places where the first code X1 is changed exceeds a predetermined threshold. Specifically, among the plurality of pieces of change data Z, change data Z having, for example, 10 or more confirmed codes different from the first code X1 is selected.

The processing unit 514 in FIG. 5 generates teacher data L from the change data Z selected by the selection unit 512 . The teacher data L is, as illustrated in FIG. 5, a combination of a first code X1, a confirmed code corresponding to the first code X1, and a second feature Y2 corresponding to the first code X1. . That is, a plurality of teacher data L are generated from any one change data Z selected by the selection unit 512 . The teacher data generation unit 51 generates N pieces of teacher data L by the processing described above.

The N pieces of teacher data L are divided into N1 pieces of teacher data L and N2 pieces of teacher data L (N=N1+N2). The N1 pieces of training data L (hereinafter referred to as "modified training data L1") include the first code X1 subject to modification by the user. That is, the confirmed code included in each of the N1 changed teacher data L1 is a code after changing the first code X1 (that is, a code different from the first code X1). The N1 pieces of changed teacher data L1 are learning big data representing the first tendency described above. On the other hand, the N2 pieces of teaching data L (hereinafter referred to as "unchanged teaching data L2") include the first code X1 that has not been subject to change by the user. That is, the confirmed code included in each of the N2 pieces of unchanged teaching data L2 is the same code as the first code X1. N teacher data L including N1 changed teacher data L1 and N2 unchanged teacher data L2 correspond to learning big data representing the second tendency.

The learning unit 53 generates the coefficient K1 and the coefficient K2 from the N pieces of teacher data L generated by the teacher data generating unit 51 . The learning section 53 of the first embodiment includes a first learning section 532 and a second learning section 534 . The first learning unit 532 generates a plurality of coefficients K1 that define the first trained model M1 by machine learning (deep learning) using N1 changed teacher data L1 out of N teacher data L. . That is, a plurality of coefficients K1 reflecting the first tendency are generated. The first trained model M1 defined by a plurality of coefficients K1 has a first code X1 and a second feature quantity Y2, and a confirmed code (second It is a prediction model that has learned the relationship between code X2).

The second learning unit 534 performs machine learning using N pieces of teacher data (N1 pieces of changed teacher data L1 and N2 pieces of unchanged teacher data L2) to obtain a plurality of coefficients that define the second learned model M2. Generate K2. That is, a plurality of coefficients K2 reflecting the second tendency are generated. A second trained model M2 defined by a plurality of coefficients K2 is a relationship between the first code X1 and the second feature quantity Y2 and the confirmed code under the tendency expressed by the N pieces of teacher data L. is a prediction model that has learned A plurality of coefficients K 1 and a plurality of coefficients K 2 generated by machine learning device 200 are stored in storage device 13 of code estimation device 100 .

FIG. 6 is a flow chart of processing (hereinafter referred to as "chord estimation processing") for estimating the second code X2 by the control device 12 of the chord estimation device 100. As shown in FIG. The code estimation process is started when the acoustic signal V transmitted from the terminal device 300 is received, for example. When the chord estimation process is started, the first extractor 21 extracts the first feature Y1 from the acoustic signal V (Sa1). The analysis unit 23 estimates the first code X1 from the first feature Y1 extracted by the first extraction unit 21 (Sa2). The second extraction unit 25 extracts the second feature Y2 from the first feature Y1 extracted by the first extraction unit 21 and the first code X1 estimated by the analysis unit 23 (Sa3). The code estimation unit 27 estimates the second code X2 by inputting the first code X1 and the second feature Y2 to the learned model M (Sa4).

FIG. 7 is a detailed flowchart of the processing (Sa4) of the chord estimator 27. As shown in FIG. The chord estimator 27 generates the appearance probability λ1 for each candidate chord using the first trained model M1 that has learned the first tendency (Sa4-1). The chord estimator 27 generates the appearance probability λ2 for each candidate chord using the second trained model M2 that has learned the second tendency (Sa4-2). The order of generating the occurrence probability λ1 (Sa4-1) and generating the occurrence probability λ2 (Sa4-2) may be reversed. The code estimation unit 27 integrates the appearance probability λ1 generated by the first trained model M1 and the appearance probability λ2 generated by the second trained model M2 for each candidate code, thereby obtaining the appearance probability λ0 for each candidate code. is calculated (Sa4-3). The code estimator 27 estimates a candidate code with a high appearance probability λ0 among the Q candidate codes as the second code X2 (Sa4-4).

As can be understood from the above description, in the first embodiment, the second code X2 is estimated by inputting the first code X1 and the second feature Y2 to the trained model M that has learned the tendency of changes to the code. Therefore, the second code X2 can be estimated with high accuracy in consideration of the change tendency of the code, as compared with the configuration of only estimating the first code X1 from the acoustic signal V. FIG.

In the first embodiment, the estimation result (occurrence probability λ1) by the first trained model M1 that has learned the first tendency and the estimation result (occurrence probability λ2) by the second trained model M2 that has learned the second tendency are Based on this, the second code X2 is estimated. For example, in the method of estimating the second code X2 based on either the estimation result of the first trained model M1 or the estimation result of the second trained model M2, the second code X2 appropriately reflects the tendency of changes to the code. The problem is that X2 is not estimated. Specifically, in the method of estimating the second code X2 based only on the estimation result of the first trained model M1, the input first code X1 is always changed. Further, in the method of estimating the second code X2 based only on the estimation result of the second trained model M2, the first code X1 is less subject to change. According to the configuration of the first embodiment in which the second code X2 is estimated using the first trained model M1 and the second trained model M2, for example, either the first trained model M1 or the second trained model M2 Compared to the method of estimating the second code X2 using either one, it is possible to estimate the second code X2 that appropriately reflects the tendency of changes to the code.

In the first embodiment, the second feature Y2 including the variance σq and average μq in the time series of the component intensity Pq and the variance σv and average μv in the time series of the intensity Pv of the acoustic signal V is inputted to the trained model. Thus, since the second code X2 is estimated, the second code X2 can be estimated with high accuracy in consideration of the time change of the acoustic signal V.

<Second embodiment>
A second embodiment of the present invention will be described. In addition, in each aspect illustrated below, the reference numerals used in the description of the first embodiment are used for the elements whose functions or actions are the same as those of the first embodiment, and the detailed description of each element is appropriately omitted. In the first embodiment, the second code X2 is estimated by inputting the first code X1 and the second feature Y2 into the trained model M, but the data input to the trained model M are exemplified below. Modified like each form.

FIG. 8 is a configuration diagram of the chord estimation unit 27 in the second embodiment. In the second embodiment, by inputting the first code X1 to the trained model M, the second code X2 is estimated. That is, the learned model M of the second embodiment is a prediction model that has learned the relationship between the first code X1 and the second code X2 (confirmed code). The first code X1 input to the trained model M is generated by the same method as in the first embodiment. In the second embodiment, extraction of the second feature quantity Y2 (second extraction unit 25) is omitted.

<Third Embodiment>
FIG. 9 is a configuration diagram of the chord estimation unit 27 in the third embodiment. In the third embodiment, the second code X2 is estimated by inputting the first feature Y1 to the learned model M. That is, the learned model M of the third embodiment is a prediction model that has learned the relationship between the first feature Y1 and the second code X2 (confirmed code). The first feature Y1 input to the trained model M is generated by the same method as in the first embodiment. In the third embodiment, the estimation of the first code X1 (analysis section 23) and the extraction of the second feature Y2 (second extraction section 25) are omitted. According to the configuration of the third embodiment in which the first feature amount Y1 is input to the learned model M, the tendency of the code to be changed by the user is taken into account. can identify the second code X2.

<Fourth Embodiment>
FIG. 10 is a configuration diagram of the code estimation unit 27 in the fourth embodiment. In the third embodiment, the second code X2 is estimated by inputting the second feature Y2 to the trained model M. FIG. That is, the learned model M of the fourth embodiment is a prediction model that has learned the relationship between the second feature Y2 and the second code X2 (confirmed code). The second feature Y2 input to the trained model M is generated by the same method as in the first embodiment.

As can be understood from the above description, the data input to the trained model M for estimating the second code X2 from the acoustic signal V is an index representing the acoustic characteristics of the acoustic signal V (hereinafter referred to as "the (referred to as “feature quantity”). As the feature amount of the acoustic signal V, any one of the first feature amount Y1, the second feature amount Y2 and the first code X1, or a combination thereof is exemplified. Note that the feature amount of the acoustic signal V is not limited to the first feature amount Y1, the second feature amount Y2, or the first code X1. For example, a frequency spectrum may be used as the feature amount of the acoustic signal V. FIG. As can be understood from the above description, the feature amount of the acoustic signal V is arbitrary as long as the feature amount reflects the code difference.

As can be understood from the above description, the trained model M is comprehensively expressed as a statistical estimation model that has learned the relationship between the feature quantity of the acoustic signal V and the code. According to the configuration of each of the above-described modes for estimating the second code X2 from the acoustic signal V by inputting the feature amount of the acoustic signal V into the trained model M, following the tendency learned by the trained model M, code is estimated. Therefore, compared to the configuration in which the code is estimated by comparing the code prepared in advance with the feature amount of the acoustic signal V (for example, the frequency spectrum in Patent Document 1), it is possible to accurately estimate the code from various feature amounts of the acoustic signal V. You can infer the code. That is, with the technique of Patent Document 1, when the feature amount of the acoustic signal V is extremely deviated from the code prepared in advance, an appropriate code cannot be estimated with high accuracy. On the other hand, according to the configuration of each of the above-described forms, the code is estimated along the tendency learned by the trained model M, so that the appropriate code can be determined with high accuracy regardless of the content of the feature amount of the acoustic signal V. can be estimated to

Of the trained models M that have learned the relationship between the feature quantity of the acoustic signal V and the chords, the trained model M that inputs the first code (for example, the trained model M illustrated in the first and second embodiments) M) is represented generically as a trained model M that has learned changes on the code.

<Fifth Embodiment>
FIG. 11 is a block diagram illustrating the functional configuration of the control device 12 in the chord estimation device 100 according to the fifth embodiment of the invention. The controller 12 of the fifth embodiment functions as a boundary estimation model Mb in addition to the same elements (the preprocessing section 20, the second extraction section 25 and the code estimation section 27) as in the first embodiment. The time series of the first feature Y1 generated by the first extraction unit 21 is input to the boundary estimation model Mb. The boundary estimation model Mb is a trained model that has learned the relationship between the time series of the first feature Y1 and the boundary data B. FIG. That is, the boundary estimation model Mb outputs boundary data B according to the time series of the first feature Y1. Boundary data B is time-series data representing the boundary of each continuous section on the time axis. A continuous section is a series of sections in which the same code in the acoustic signal V continues. For example, a recurrent neural network (RNN: Recurrent Neural Network) such as a long short term memory (LSTM) suitable for processing time series data is preferably used as the boundary estimation model Mb.

12 is an explanatory diagram of the boundary data B. FIG. The boundary data B includes a time series of unit data b corresponding to each unit period T on the time axis. One unit data b is output from the boundary estimation model Mb for each first feature amount Y1 in each unit period T. FIG. The unit data b corresponding to each unit period T is binary data representing whether or not the time point corresponding to the unit period T corresponds to the boundary of the continuous section. For example, the unit data b is set to a numerical value of 1 when the starting point of the unit period T is the boundary of the continuous section, and is set to a numerical value of 0 when the starting point of the unit period T does not correspond to the boundary of the continuous section. That is, the numerical value 1 of the unit data b means that the unit period T corresponding to the unit data b is the beginning of the continuous section. As understood from the above description, the boundary estimation model Mb is a statistical estimation model that estimates the boundary of each continuous section from the time series of the first feature Y1. Boundary data B is time-series data that binary indicates whether or not each of a plurality of time points on the time axis corresponds to a boundary of a continuous section.

The boundary estimation model Mb includes a program (for example, a program module constituting artificial intelligence software) that causes the control device 12 to execute a calculation for generating the boundary data B from the time series of the first feature Y1, and a plurality of is realized in combination with the coefficient Kb of A plurality of coefficients Kb are set by machine learning (especially deep learning) using a plurality of teacher data Lb and stored in the storage device 13 .

The second extraction unit 25 of the first embodiment extracts the second feature Y2 for each continuous interval, which is the continuous interval where the first code X1 analyzed by the analysis unit 23 is continuous. The second extraction unit 25 of the fifth embodiment extracts the second feature Y2 for each continuous section represented by the boundary data B output from the boundary estimation model Mb. Specifically, the second extraction unit 25 generates a second feature Y2 from one or more first features Y1 in the continuous section represented by the boundary data B. FIG. Therefore, the input of the first code X1 to the second extractor 25 is omitted. The content of the second feature Y2 is the same as in the first embodiment.

FIG. 13 is a flowchart illustrating a specific procedure of chord estimation processing in the fifth embodiment. When the chord estimation process is started, the first extractor 21 extracts the first feature Y1 from the acoustic signal V for each unit period T (Sb1). The analysis unit 23 estimates the first code X1 from the first feature Y1 extracted by the first extraction unit 21 for each unit period T (Sb2).

The boundary estimation model Mb generates boundary data B from the time series of the first feature Y1 extracted by the first extraction unit 21 (Sb3). The second extraction unit 25 extracts the second feature Y2 from the first feature Y1 extracted by the first extraction unit 21 and the boundary data B generated by the boundary estimation model Mb (Sb4). Specifically, for each continuous section represented by the boundary data B, the second extraction unit 25 generates the second feature Y2 from one or more first feature Y1 in the continuous section. The code estimation unit 27 estimates the second code X2 by inputting the first code X1 and the second feature Y2 to the trained model M (Sb5). A specific procedure for estimating the second code X2 (Sb5) is the same as in the first embodiment (FIG. 7). The order of the estimation of the first code X1 by the analysis unit 23 (Sb2) and the generation of the boundary data B by the boundary estimation model Mb (Sb3) may be reversed.

FIG. 14 is a block diagram illustrating the configuration of a machine learning device 200 that sets multiple coefficients Kb of the boundary estimation model Mb. A machine learning device 200 according to the fifth embodiment includes a third learning unit 55 . The third learning unit 55 sets a plurality of coefficients Kb by machine learning using a plurality of teacher data Lb. As illustrated in FIG. 14, each of the plurality of teacher data Lb includes the time series of the first feature Y1 and the boundary data Bx. The boundary data Bx is composed of a time series (that is, correct values) of known unit data b corresponding to each first feature Y1. That is, among the plurality of unit data b of the boundary data Bx, the unit data b corresponding to the unit period T at the beginning of each continuous section is set to a numerical value of 1, and the unit data b corresponding to the unit periods T other than the beginning of each continuous section is set. Data b is set to a numerical value of zero.

The third learning unit 55 inputs the time series of the first feature Y1 of the teacher data Lb, and combines the boundary data B output from the temporary boundary estimation model Mb with the boundary data Bx of the teacher data Lb. Update the coefficients Kb of the boundary estimation model Mb so that the discrepancies are reduced. Specifically, the third learning unit 55 iteratively updates a plurality of coefficients Kb by, for example, error backpropagation so that the evaluation function representing the difference between the boundary data B and the boundary data Bx is minimized. . A plurality of coefficients Kb set by the machine learning device 200 in the above procedure are stored in the storage device 13 of the code estimation device 100 . Therefore, the boundary estimation model Mb is based on the latent tendency between the time series of the first feature Y1 in the plurality of teacher data Lb and the boundary data Bx, for the time series of the unknown first feature Y1 output statistically valid boundary data B. Note that the third learning unit 55 may be installed in the chord estimation device 100 .

As described above, according to the fifth embodiment, the boundary data B related to the unknown acoustic signal V is calculated using the boundary estimation model Mb that has learned the relationship between the time series of the first feature Y1 and the boundary data B. is generated. Therefore, by using the second feature Y2 generated according to the boundary data B, it is possible to estimate the second code X2 with high accuracy.

<Sixth Embodiment>
FIG. 15 is a block diagram illustrating the functional configuration of the control device 12 in the chord estimation device 100 according to the sixth embodiment of the invention. The chord estimation unit 27 of the sixth embodiment includes a chord transition model Mc in addition to the same elements (learned model M and estimation processing unit 70) as in the first embodiment. The time series of the second feature Y2 output by the second extractor 25 is input to the chord transition model Mc. The chord transition model Mc is a trained model that has learned chord transition tendencies. The tendency of chord transitions is, for example, chord arrangements that tend to appear in a large number of existing songs. Specifically, the chord transition model Mc is a learned model that has learned the relationship between the time series of the second feature Y2 and the time series of the code data C representing the chord. That is, the chord transition model Mc outputs the chord data C according to the time series of the second feature Y2 for each continuous section. For example, a recurrent neural network (RNN) such as a long short-term memory (LSTM) suitable for processing time-series data is preferably used as the chord transition model Mc.

The code data C of the sixth embodiment represents the appearance probability λc for each of Q candidate codes. The appearance probability λc corresponding to any one candidate code means the probability (or likelihood) that the code in the continuous section of the acoustic signal V corresponds to the candidate code. The appearance probability λc is set to a numerical value within the range of 0 or more and 1 or less. As can be understood from the above description, the time series of code data C represents code transitions. That is, the chord transition model Mc is a statistical estimation model for estimating chord transitions from the time series of the second feature Y2.

The estimation processing unit 70 of the sixth embodiment generates the appearance probability λ1 output by the first trained model M1, the appearance probability λ2 output by the second trained model M2, and the chord data C output by the chord transition model Mc. Estimate the second code X2 based on Specifically, the estimation processing unit 70 calculates the appearance probability λ0 of each candidate code by integrating the appearance probability λ1, the appearance probability λ2, and the appearance probability λc of the code data C for each candidate code. The occurrence probability λ0 of each candidate code is, for example, the weighted sum of the occurrence probability λ1, the occurrence probability λ2, and the occurrence probability λc. The estimation processing unit 70 estimates a candidate code having a high appearance probability λ0 among the Q candidate codes for each unit period T as the second code X2. As can be understood from the above description, in the sixth embodiment, the second code X2 is generated based on the output of the trained model M (that is, the occurrence probability λ1 and the occurrence probability λ2) and the code data C (the occurrence probability λc). Presumed. That is, in addition to the above-described first and second tendencies, the second chord X2 is estimated in consideration of the chord transition tendency learned by the chord transition model Mc.

The code transition model Mc is a program (for example, a program module constituting artificial intelligence software) that causes the control device 12 to execute a calculation for generating a time series of code data C from a time series of the second feature Y2, and is applied to the calculation. is realized in combination with a plurality of coefficients Kc that are A plurality of coefficients Kc are set by machine learning (especially deep learning) using a plurality of teacher data Lc and stored in the storage device 13 .

FIG. 16 is a flowchart illustrating a specific procedure of processing (Sa4) for estimating the second code X2 by the code estimator 27 of the sixth embodiment. In the sixth embodiment, step Sa4-3 in the processing of the first embodiment described with reference to FIG. 7 is replaced with steps Sc1 and Sc2 in FIG.

When the occurrence probability λ1 and the occurrence probability λ2 are generated for each candidate code (Sa4-1, Sa4-2), the code estimation unit 27 converts the time series of the second feature Y2 extracted by the second extraction unit 25 into a code transition model A time series of code data C is generated by inputting to Mc (Sc1). The order of generating the occurrence probability λ1 (Sa4-1), generating the occurrence probability λ2 (Sa4-2), and generating the code data C (Sc1) can be arbitrarily changed.

The code estimator 27 integrates the appearance probabilities λ1 and λ2 and the appearance probabilities λc represented by the code data C for each candidate code to calculate the appearance probability λ0 for each candidate code (Sc2). The code estimator 27 estimates a candidate code with a high appearance probability λ0 among the Q candidate codes as the second code X2 (Sa4-4). The specific procedure of the process of estimating the second code X2 in the sixth embodiment is as described above.

FIG. 17 is a block diagram illustrating the configuration of a machine learning device 200 that sets multiple coefficients Kc of the chord transition model Mc. A machine learning device 200 according to the sixth embodiment includes a fourth learning unit 56 . The fourth learning unit 56 sets a plurality of coefficients Kc by machine learning using a plurality of teacher data Lc. As illustrated in FIG. 17, each of the plurality of teacher data Lc includes a time series of the second feature Y2 and a time series of the code data Cx. The chord data Cx is composed of Q occurrence probabilities λc corresponding to different candidate chords, and is generated according to chord transitions in a known piece of music. That is, out of the Q occurrence probabilities λc of the chord data Cx, the occurrence probability λc corresponding to one candidate chord that actually appears in a known piece of music is set to 1, and the remaining (Q−1) occurrence probabilities λc The probability of occurrence λc corresponding to the candidate code is set to zero.

The fourth learning unit 56 inputs the time series of the second feature Y2 of the teacher data Lc, and the time series of the code data C output from the provisional chord transition model Mc, and the code data of the teacher data Lc. A plurality of coefficients Kc of the chord transition model Mc are updated so as to reduce discrepancies with the time series of Cx. Specifically, the fourth learning unit 56 calculates a plurality of coefficients Kc by, for example, error backpropagation so that the evaluation function representing the difference between the time series of the code data C and the time series of the code data Cx is minimized. is iteratively updated. A plurality of coefficients Kc set by the machine learning device 200 in the above procedure are stored in the storage device 13 of the code estimation device 100 . Therefore, the chord transition model Mc is a latent tendency between the time series of the second feature Y2 and the time series of the chord data Cx in the plurality of teacher data Lc (that is, the tendency of chord transitions appearing in existing songs). , a time series of code data C that is statistically valid for the time series of the unknown second feature Y2 is output. Note that the fourth learning unit 56 may be installed in the chord estimation device 100 .

As described above, according to the sixth embodiment, using the chord transition model Mc that has learned the relationship between the time series of the second feature Y2 and the time series of the code data C, A second code X2 is estimated. Therefore, compared to the first embodiment that does not use the chord transition model Mc, it is possible to estimate the second chords X2 with perceptually natural sequences that are used in many songs. Note that the boundary estimation model Mb may be omitted in the sixth embodiment.

<Modification>
Specific modified aspects added to the above-exemplified aspects will be exemplified below. Two or more aspects arbitrarily selected from the following examples may be combined as appropriate within a mutually consistent range.

(1) In the above embodiments, the code estimation device 100 is separate from the terminal device 300 of the user U, but the code estimation device 100 may be installed in the terminal device 300 . According to the configuration in which the terminal device 300 and the code estimation device 100 are integrated, it becomes unnecessary to transmit the acoustic signal V to the code estimation device 100 . However, according to the above-described configurations in which terminal device 300 and code estimation device 100 are separate entities, the processing load on terminal device 300 is reduced. Elements for extracting the feature quantity of the acoustic signal V (for example, the first extraction unit 21 , the analysis unit 23 and the second extraction unit 25 ) may be installed in the terminal device 300 . The terminal device 300 transmits the feature quantity of the acoustic signal V to the code estimation device 100 , and the code estimation device 100 transmits to the terminal device 300 the second code X 2 estimated from the feature quantity transmitted from the terminal device 300 .

(2) In each of the above embodiments, the trained model M is composed of the first trained model M1 and the second trained model M2, but the mode of the trained model M is not limited to the above examples. For example, using N pieces of teacher data L, a statistical estimation model that has learned the first tendency and the second tendency may be used as the learned model M. The learned model M outputs the appearance probability for each code based on, for example, the first trend and the second trend. That is, the process of calculating the appearance probability λ0 in the estimation processing unit 70 can be omitted.

(3) In each of the above embodiments, the second trained model M2 learns the second tendency, but the tendency learned by the second trained model M2 is not limited to the above examples. For example, the second trained model M2 may learn only whether or not the code has been changed. That is, the first trend does not have to be included in the second trend.

(4) In each of the above forms, the trained model (M1, M2) outputs the occurrence probability (λ1, λ2) of each code, but the data output by the trained model M is the occurrence probability (λ1, λ2) is not limited to For example, the first trained model M1 and the second trained model M2 may output the code itself.

(5) In each of the above embodiments, one second code X2 corresponding to the first code X1 is estimated, but a plurality of second codes X2 corresponding to the first code X1 may be estimated. Among the occurrence probabilities λ0 of the codes calculated by the estimation processing unit 70, a plurality of codes having higher occurrence probabilities λ0 may be transmitted to the terminal device 300 as the second codes X2. The user U specifies a desired code from the plurality of transmitted second codes X2.

(6) In each of the above-described forms, the feature amount corresponding to any one unit period T is input to the learned model M, but together with the feature amount corresponding to any one unit period T, the unit period You may input the feature-value of the front or back of T to the learned model M.

(7) In each of the above embodiments, the chroma vector including a plurality of component intensities Pq corresponding to each of a plurality of tones of the scale and the first feature Y1 including the intensity Pv of the acoustic signal V were exemplified. The contents of the quantity Y1 are not limited to the above examples. For example, a chroma vector may be used as the first feature Y1. Further, the variance σq and the average μq regarding the time series of the component strength Pq for each tone of the scale indicated by the chroma vector may be used as the second feature amount Y2. The contents of the first feature amount Y1 and the second feature amount Y2 are arbitrary as long as the feature amount reflects the code difference.

(8) In each of the above embodiments, the chord estimation apparatus 100 estimates the second code X2 from the feature quantity of the acoustic signal V using the trained model M, but the method of estimating the second code X2 is limited to the above examples. not. For example, the code associated with the second feature quantity Y2 most similar to the second feature quantity Y2 extracted by the second extraction unit 25 among the plurality of second feature quantities Y2 associated with different codes is selected as the first It may be estimated as 2 codes X2.

(9) In the above-described fifth embodiment, the boundary data B binary representing whether or not each unit period T corresponds to the boundary of the continuous section was exemplified. Not limited. For example, the boundary estimation model Mb may output boundary data B representing the likelihood that each unit period T is the boundary of continuous sections. Specifically, each unit data b of the boundary data B is set to a numerical value within the range of 0 or more and 1 or less, and the sum of the numerical values represented by the plurality of unit data b becomes a predetermined value (eg, 1). The second extraction unit 25 estimates the boundary of the continuous section from the likelihood represented by each unit data b of the boundary data B, and extracts the second feature Y2 for each continuous section.

(10) In the sixth embodiment described above, the chord transition model Mc that learns the relationship between the time series of the second feature Y2 and the time series of the code data C was exemplified. The quantity is not limited to the second feature quantity Y2. For example, in a configuration in which the chord transition model Mc learns the relationship between the time series of the first feature Y1 and the time series of the code data C, the time series of the first feature Y1 extracted by the first extraction unit 21 is code transition Input to the model Mc. The code transition model Mc outputs a time series of code data C according to the time series of the first feature Y1. A code transition model Mc that has learned the relationship between the time series of the feature quantity different from the first feature quantity Y1 and the second feature quantity Y2 and the time series of the code data C is used for estimating the time series of the code data C. You may

(11) In the sixth embodiment described above, the code data C representing the occurrence probability λc of 0 or more and 1 or less for each of the Q candidate codes was exemplified. is not limited to For example, the chord transition model Mc outputs code data C in which the appearance probability λc of any one of Q candidate codes is set to 1 and the remaining (Q-1) appearance probabilities λc are set to 0. You may That is, the code data C is a Q-dimensional vector that expresses one of the Q candidate codes in a one-hot format.

(12) In the sixth embodiment, the chord estimation device 100 including the learned model M, the boundary estimation model Mb, and the chord transition model Mc is illustrated. may be used. For example, in an information processing device (boundary estimation device) that estimates the boundary of each continuous section from the time series of the first feature Y1 using the boundary estimation model Mb, the trained model M and the chord transition model Mc are not essential. . In the information processing device (chord transition estimation device) that estimates the code data C from the time series of the second feature Y2 using the chord transition model Mc, the trained model M and the boundary estimation model Mb are not essential. Also, the learned model M is omitted in the information processing apparatus having the boundary estimation model Mb and the chord transition model Mc. That is, generation of the occurrence probability λ1 and the occurrence probability λ2 is not essential. For example, among Q candidate codes, a candidate code having a high appearance probability λc output by the code transition model Mc is output as the second code X2 every unit period T.

(13) The code estimation device 100 and the machine learning device 200 according to each of the above embodiments are implemented by cooperation between a computer (specifically, a control device) and a program, as illustrated in each embodiment. The program according to each of the forms described above can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disc) such as a CD-ROM is a good example. may include a recording medium in the form of The non-transitory recording medium includes any recording medium other than transitory, propagating signals, and does not exclude volatile recording media. It is also possible to provide the computer with the program in the form of distribution via a communication network. Further, the execution body of the program is not limited to the CPU, and the program may be executed by a neural network processor such as a Tensor Processing Unit and a Neural Engine, or a DSP (Digital Signal Processor) for signal processing. In addition, multiple types of subjects selected from the above examples may work together to execute the program.

(14) A trained model (first trained model M1, second trained model M2, boundary estimation model Mb, or code transition model Mc) is a statistical estimation model (for example, A neural network) that produces an output B in response to an input A. Specifically, the trained model consists of a program (for example, a program module that constitutes artificial intelligence software) that causes the control device to execute a calculation that specifies output B from input A, and a plurality of coefficients that are applied to the calculation. Realized in combination. A plurality of coefficients of the trained model are optimized by prior machine learning (deep learning) using a plurality of teacher data in which the input A and the output B are matched. That is, the trained model is a statistical estimation model that has learned the relationship between the input A and the output B. The control device performs calculations applying a plurality of learned coefficients and a predetermined response function to an unknown input A, thereby detecting tendencies latent in a plurality of teacher data (between input A and output B produces a statistically valid output B for an input A under the relationship ).

(15) For example, the following configurations can be grasped from the above-exemplified forms.

A chord estimation method according to a preferred aspect (first aspect) of the present invention estimates a first chord from an acoustic signal, inputs the first chord to a trained model that has learned a tendency of changes to the chord, Estimate the second code. According to the above aspect, since the second code is estimated by inputting the first code estimated from the acoustic signal to the trained model that has learned the tendency of changes to the chord, the first code is estimated from the acoustic signal. Compared to the configuration of only one, it is possible to highly accurately estimate the second code that takes into account the tendency of changes to the code.

In a preferred example of the first mode (second mode), the trained models include a first trained model that has learned the tendency of changes to the code and a second trained model that has learned the tendency of changes to the code. and a model, wherein, in estimating the second code, an output when the first code is input to the first trained model and an output when the first code is input to the second trained model. and the second code is estimated according to. According to the above aspect, compared to the method of estimating the second code using either one of the first trained model and the second trained model, for example, the second code appropriately reflects the tendency of changes to the code. 2 codes can be estimated.

In a preferred example of the first aspect (third aspect), in estimating the first code, a first feature amount including, for each tone of the scale, a component intensity corresponding to the intensity of a component corresponding to a tone of the scale in the acoustic signal. , and in estimating the second code, the learned Estimate the second code by inputting it into the model. According to the above aspect, the second code is estimated by inputting the second feature amount including the variance and average regarding the time series of the component intensity of each scale sound to the trained model. can be taken into account to estimate the second code with high precision.

In a preferred example of the third aspect (fourth aspect), the first feature quantity includes the intensity of the acoustic signal, and the second feature quantity includes the variance and mean of the intensity of the acoustic signal over time. According to the above aspect, the effect that the second code can be estimated with high accuracy in consideration of the time change of the acoustic signal is particularly remarkable.

In the preferred example of the first aspect (fifth aspect), a boundary estimation model that has learned the relationship between the time series of the first feature amount and the boundary data representing the boundary of the continuous section where the code continues is applied to the first Boundary data is estimated by inputting a time series of feature amounts, a second feature amount is extracted from the time series of the first feature amount for each continuous interval represented by the boundary data, and in estimating the second code, and estimating a second code by inputting the first code and the second feature into the trained model. In the above aspect, the boundary data regarding the unknown acoustic signal is generated using the boundary estimation model that has learned the relationship between the time series of the first feature amount and the boundary data. Therefore, it is possible to estimate the second code with high accuracy by using the second feature amount generated according to the boundary data.

In the preferred embodiment of the first aspect (sixth aspect), inputting the time series of the feature amount of the acoustic signal to a chord transition model that has learned the relationship between the time series of the feature amount and the time series of the code data representing the chord. and estimates the second code based on the output of the trained model and the code data. According to the above aspect, the second code relating to the unknown acoustic signal is estimated using the code transition model that has learned the relationship between the time series of the feature amount and the time series of the code data. Therefore, compared to a configuration that does not use a chord transition model, it is possible to estimate perceptually natural sequences of second chords observed in many pieces of music.

In the preferred example (seventh aspect) of the first to sixth aspects (seventh aspect), the acoustic signal is received from a terminal device, and the second code estimated from the acoustic signal is input to the trained model. A code is estimated and the second code is transmitted to the terminal device. According to the above aspect, the processing load on the terminal device is reduced compared to, for example, a method of estimating the code using a trained model installed in the user's terminal device.

A preferred aspect of the present invention is also implemented as a chord estimation device that executes the chord estimation method of each aspect illustrated above, or a program that causes a computer to execute the chord estimation method of each aspect illustrated above. For example, a chord estimating device according to a preferred aspect of the present invention inputs the first chord to a first chord estimating unit that estimates a first chord from an acoustic signal, and a trained model that has learned the tendency of changes to chords. and a second code estimator for estimating the second code.

DESCRIPTION OF SYMBOLS 100... Code|cord estimation apparatus, 200... Machine-learning apparatus, 300... Terminal device, 11... Communication apparatus, 12... Control apparatus, 13... Storage device, 20... Pre-processing part, 21... First extraction part, 23... Analysis part, 25 Second extraction unit 27 Chord estimation unit 51 Teacher data generation unit 512 Selection unit 514 Processing unit 53 Learning unit 532 First learning unit 534 Second learning unit 55 Third learning unit 56 Fourth learning unit 70 Estimation processing unit M Learned model M1 First learned model M2 Second learned model Mb Boundary estimation model Mc Code transition model.

Claims (16)

estimating a first code from the acoustic signal;
A computer- implemented chord estimation method for estimating a second chord by inputting the first chord into a trained model that has learned a tendency of changes to the chord, comprising:
The learned model is
a first trained model that has learned trends in changes to code;
a second trained model that has learned trends in the presence or absence of changes to the code;
In estimating the second code, according to the output when the first code is input to the first trained model and the output when the first code is input to the second trained model Estimate the second code
Code estimation method. estimating a first code from the acoustic signal;
estimating a second code by inputting the first code and a second feature quantity of the acoustic signal to a trained model that has learned the tendency of changes to the code;
The second feature amount includes, for each of a plurality of scale sounds, an index related to time change of component intensity corresponding to the intensity of the component corresponding to the scale sound in the acoustic signal.
A computer-implemented code estimation method. In estimating the first code, the first code is estimated from a first feature amount including the component strength for each note of the scale.
3. The code estimation method of claim 2. The first feature amount includes the intensity of the acoustic signal,
4. The chord estimation method according to claim 3, wherein said second feature quantity includes an index relating to temporal change in intensity of said acoustic signal. estimating a first code from the acoustic signal;
Boundary data is obtained by inputting the time series of the first feature amount of the acoustic signal to a boundary estimation model that has learned the relationship between the time series of the first feature amount and boundary data representing the boundary of the continuous section where the code continues. presume,
generating a second feature value from the time series of the first feature value for each continuous section represented by the boundary data;
A chord estimation method implemented by a computer, comprising: estimating a second chord by inputting the first chord and the second feature quantity into a trained model that has learned a tendency of changes to the chord. estimating a first code from the acoustic signal;
estimating the time series of the code data by inputting the time series of the feature amount of the acoustic signal to a code transition model that has learned the relationship between the time series of the feature amount and the time series of the code data representing the code;
A chord estimation method implemented by a computer , comprising: estimating a second chord based on an output when the first chord is input to a trained model that has learned a tendency of changes to the chord, and the chord data . a first code estimation unit that estimates a first code from the acoustic signal;
a second code estimating unit for estimating a second code by inputting the first code to a trained model that has learned the tendency of changes to the code ,
The learned model is
a first trained model that has learned trends in changes to code;
a second trained model that has learned trends in the presence or absence of changes to the code;
The second code estimating unit performs the above-described A code estimation device for estimating a second code . a first code estimation unit that estimates a first code from the acoustic signal;
a second code estimator for estimating a second code by inputting the first code and the second feature quantity of the acoustic signal to a trained model that has learned the tendency of changes to the code ,
The second feature quantity includes, for each of a plurality of scale sounds, an index relating to time change of component intensity corresponding to the intensity of the component corresponding to the scale sound in the acoustic signal.
Code estimator. The first chord estimating unit estimates the first chord from a first feature amount including the component strength for each note of the scale.
9. The chord estimation apparatus of claim 8. The first feature amount includes the intensity of the acoustic signal,
10. The chord estimating apparatus according to claim 9 , wherein said second feature quantity includes an index relating to temporal change in strength of said acoustic signal. a first code estimation unit that estimates a first code from the acoustic signal;
A boundary estimation model that learns the relationship between the time series of the first feature amount and boundary data representing the boundary of the continuous section where the code continues, wherein the boundary data is obtained by inputting the time series of the first feature amount of the acoustic signal. a boundary estimation model that outputs
an extraction unit that extracts a second feature from the time series of the first feature for each continuous section represented by the boundary data;
A chord estimation device comprising: a second chord estimation unit that estimates a second chord by inputting the first chord and the second feature quantity into a trained model that has learned the tendency of chord changes. a first code estimation unit that estimates a first code from the acoustic signal;
A code transition model that learns the relationship between the time series of the feature amount and the time series of code data representing the code, wherein the time series of the feature amount of the acoustic signal is input and the time series of the code data is output. and,
a second chord estimation unit for estimating a second chord based on the chord data and an output when the first chord is input to a trained model that has learned the tendency of chord changes. . a first code estimator for estimating a first code from an acoustic signal; and
a second code estimating unit for estimating a second code by inputting the first code to a trained model that has learned the tendency of changes to the code;
A program that causes a computer to function as
The learned model is
a first trained model that has learned trends in changes to code;
a second trained model that has learned trends in the presence or absence of changes to the code;
The second code estimating unit performs the above-described guess the second chord
program. a first code estimator for estimating a first code from an acoustic signal; and
a second code estimator for estimating a second code by inputting the first code and the second feature quantity of the acoustic signal to a trained model that has learned the tendency of changes to the code;
A program that causes a computer to function as
The second feature amount includes, for each of a plurality of scale sounds, an index related to time change of component intensity corresponding to the intensity of the component corresponding to the scale sound in the acoustic signal.
program. a first code estimation unit that estimates a first code from the acoustic signal;
A boundary estimation model that learns the relationship between the time series of the first feature amount and boundary data representing the boundary of the continuous section where the code continues, wherein the boundary data is obtained by inputting the time series of the first feature amount of the acoustic signal. A boundary estimation model that outputs
an extraction unit that extracts a second feature value from the time series of the first feature value for each continuous section represented by the boundary data;
a second code estimating unit for estimating a second code by inputting the first code and the second feature quantity to a trained model that has learned the tendency of changes to the code;
A program that makes a computer function as a a first code estimation unit that estimates a first code from the acoustic signal;
A code transition model that learns the relationship between the time series of the feature amount and the time series of code data representing the code, wherein the time series of the feature amount of the acoustic signal is input and the time series of the code data is output. ,and,
a second chord estimating unit for estimating a second chord based on the chord data and an output when the first chord is input to a trained model that has learned the tendency of chord changes;
A program that makes a computer function as a

Images