JP2017161572A - Sound signal processing method and sound signal processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To determine a similarity between an input sound and a reference sound in consideration of a tone color.SOLUTION: A sound signal processing method according to one embodiment comprises the steps of: obtaining an input sound signal; computing, with NMF, a first matrix that corresponds to an amplitude spectrogram of the input sound signal and includes a first component related to a frequency and a second component related to time; obtaining a second matrix that corresponds to an amplitude spectrogram of a reference sound signal, includes the first component and the second component, and is computed with the NMF; computing, for each of the second components, a similarity of a combination of the first components in the first matrix and the second matrix; and integrating, regarding the second components, similarities of combinations of the first components and obtaining a tone color similarity regarding the input sound signal and the reference sound signal.SELECTED DRAWING: Figure 14

Description

  The present invention relates to a technique for analyzing a sound signal of a music piece.

  A technique for analyzing the sound signal of music is known. For example, Patent Document 1 describes a technique for analyzing the genre and style of a song using nonnegative matrix factorization (NMF).

JP-A-2015-79110

  In Patent Document 1, the similarity between an input sound and a reference sound is determined based on a rhythm pattern. In contrast, the present invention provides a technique for determining the similarity between the input sound and the reference sound in consideration of the timbre.

  According to the present invention, an input sound signal is obtained, and a first matrix corresponding to an amplitude spectrogram of the input sound signal and including a first component related to frequency and a second component related to time is obtained by a predetermined algorithm. Obtaining a second matrix corresponding to an amplitude spectrogram of a reference sound signal, including the first component and the second component, and calculated by the predetermined algorithm; the first matrix and the Calculating the similarity of the combination of the first components in the second matrix for each of the second components, integrating the similarity of the combination of the first components for the second component, and calculating the input sound signal and the reference Obtaining a first similarity related to the tone color of the sound signal.

  In this sound signal processing method, a similarity in time change in a specific first component of the first matrix and the second matrix is calculated, and a second similarity regarding a rhythm of the input sound signal and the reference sound signal is calculated. And calculating a third similarity obtained by integrating the first similarity and the second similarity.

  The sound signal processing method includes a step of calculating a similarity between a beat spectrum of the input sound signal and a beat spectrum of the reference sound signal to obtain a fourth similarity related to a rhythm, and in the calculation of the third similarity The first similarity, the second similarity, and the fourth similarity may be integrated.

  In the calculation of the third similarity, the first similarity, the second similarity, and the weighting calculation adjusted so that the ratio between the similarity related to the timbre and the similarity related to the rhythm is 1: 1. The fourth similarity may be integrated.

  The present invention also provides a first acquisition unit that acquires an input sound signal, and a first matrix that includes a first component related to frequency and a second component related to time, corresponding to an amplitude spectrogram of the input sound signal. An observation matrix calculation means for calculating by a predetermined algorithm, a reference corresponding to an amplitude spectrogram of a reference sound signal, including the first component and the second component, and obtaining a second matrix calculated by the predetermined algorithm Matrix acquisition means, combination similarity calculation means for calculating the similarity of the combination of the first components in the first matrix and the second matrix for each of the second components, and the similarity of the combination of the first components A tone signal processing unit that integrates the second component and calculates a first similarity degree relating to the tone color of the input tone signal and the reference tone signal. To provide a device.

  According to the present invention, the similarity between the input sound and the reference sound can be determined in consideration of the timbre.

The figure which illustrates functional composition of music search system 1 concerning one embodiment. The figure which illustrates the function structure of the music search system. The figure which illustrates the detailed functional structure of the specific means 12. The figure which illustrates the detailed functional structure of the similarity calculation means 13. The figure which illustrates the detailed functional structure of the similarity calculation means 15. 2 is a diagram illustrating a hardware configuration of the electronic musical instrument 10. FIG. The figure which illustrates the hardware constitutions of the information processing apparatus. The flowchart which shows the outline | summary of operation | movement of the music search system 1. FIG. The flowchart which shows the detail of an object area specific process. The flowchart which illustrates the detail of music structure analysis. The figure which illustrates the music structure specified with respect to the input sound signal. The flowchart which illustrates the detail of object area selection processing. The figure which shows the outline | summary of NMF with respect to an amplitude spectrogram. The flowchart which illustrates the detail of the similarity calculation by NMF. The figure which illustrates the combination of a base. The flowchart which illustrates the detail of the similarity calculation by a beat spectrum. The figure which illustrates a beat spectrum.

1. Configuration FIG. 1 is a diagram illustrating a functional configuration of a music search system 1 according to an embodiment. The music search system 1 stores a plurality of music data in advance. When an input of a sound of a music to be processed (a music to be a search key) is accepted (hereinafter, this sound is referred to as an “input sound”, and a signal indicating the input sound is referred to as an “input sound signal”), a music search system 1 searches for music similar to the input sound from the stored music.

  In this example, the music search system 1 includes an electronic musical instrument 10 and an information processing device 20. The electronic musical instrument 10 is an example of a music storage device that stores music data to be searched. The information processing apparatus 20 is an example of a user terminal that provides a user interface. The music data stored in the electronic musical instrument is music data for accompaniment (hereinafter, this data is referred to as “accompaniment data”, and the sound of the music for accompaniment is referred to as “accompaniment sound”). For example, the user inputs information on the music that he / she intends to perform to the information processing apparatus 20. The music information is, for example, a sound signal of music based on sound data in an uncompressed or compressed format (wav, mp3, etc.), but is not limited to this. In addition, the music information may be stored in advance in the storage 203 of the information processing apparatus 20 to be described later, or may be input from outside the information processing apparatus 20. The information processing apparatus 20 searches the accompaniment data stored in the electronic musical instrument for data similar to the input music. When the accompaniment sound similar to the input music is found, the information processing apparatus 20 instructs the electronic musical instrument 10 to reproduce the accompaniment sound. The electronic musical instrument 10 reproduces the instructed accompaniment sound. The user plays the electronic musical instrument 10 in accordance with the accompaniment to be played.

  FIG. 2 is a diagram illustrating a functional configuration of the music search system 1. When a music sound signal (input sound signal) is input, the music search system 1 outputs a music similar to the music. The music search system 1 includes an acquisition unit 11, an identification unit 12, a similarity calculation unit 13, a database 14, a similarity calculation unit 15, an integration unit 16, a selection unit 17, and an output unit 18.

  The acquisition unit 11 acquires an input sound signal. The specifying unit 12 specifies a target section that is a target of subsequent processing in the input sound signal. The database 14 stores information related to a plurality of accompaniment data. The similarity calculation means 13 calculates the similarity between the input sound and the accompaniment sound by using nonnegative matrix factorization (NMF) in the target section of the input sound signal. The similarity calculation means 15 calculates the similarity between the input sound and the accompaniment sound using the beat spectrum in the target section of the input sound signal. The integration unit 16 integrates the similarity calculated by the similarity calculation unit 13 and the similarity calculated by the similarity calculation unit 15. The selection means 17 selects the music similar to the input sound from the database 14 based on the integrated similarity. The output means 18 outputs the selected music piece.

  FIG. 3 is a diagram illustrating a detailed functional configuration of the specifying unit 12. The specifying unit 12 outputs a sound signal excluding a portion other than the target section (hereinafter referred to as “non-target section”) with respect to the input sound signal. The specifying unit 12 includes a structure analyzing unit 121, a dividing unit 122, a selecting unit 123, and a signal generating unit 124. The structure analysis unit 121 analyzes the musical structure of the music indicated by the input sound signal (hereinafter referred to as “music structure analysis”). The dividing unit 122 divides the input sound signal into a plurality of sections in the time domain according to the result of the music structure analysis. The selection unit 123 selects a section to be a target section from among a plurality of sections. The signal generation unit 124 generates a sound signal obtained by removing the non-target section from the input sound signal, that is, a sound signal only for the target section.

  FIG. 4 is a diagram illustrating a detailed functional configuration of the similarity calculation unit 13. The similarity calculation means 13 outputs a similarity relating to the timbre (hereinafter referred to as “tone similarity”) and a similarity relating to the rhythm (hereinafter referred to as “rhythm similarity”) with respect to the input sound signal. The similarity calculation unit 13 includes an observation matrix calculation unit 131, a reference matrix acquisition unit 132, a combination similarity calculation unit 133, a timbre similarity calculation unit 134, and a rhythm similarity calculation unit 135. The observation matrix calculation means 131 converts a matrix corresponding to the amplitude spectrogram of the input sound signal (hereinafter referred to as “observation matrix”) into a base matrix and an activation by a predetermined algorithm (in this example, NMF, details of NMF will be described later). Decomposes a matrix (coefficient matrix) product. Hereinafter, the basis matrix and the activation matrix obtained from the input sound signal are referred to as “observation basis matrix” and “observation activation matrix”, respectively. The observation base matrix is an example of a first matrix that corresponds to the amplitude spectrogram of the input sound signal and includes a first component related to frequency and a second component related to time. The reference matrix acquisition unit 132 acquires a base matrix and an activation matrix obtained by NMF from the reference sound signal. Hereinafter, the basis matrix and the activation matrix obtained from the reference sound signal are referred to as “reference basis matrix” and “reference activation matrix”, respectively. The reference sound signal is a sound signal indicating a reference music. The reference music is a music indicated by one accompaniment data sequentially selected from the accompaniment data recorded in the database 14. The reference basis matrix corresponds to the amplitude spectrogram of the reference sound signal, includes a first component and a second component, and is an example of a second matrix calculated by the above-described predetermined algorithm. The combination similarity calculation unit 133 calculates the similarity of combinations of bases included in the observation base matrix and the reference base matrix for each unit time. The timbre similarity calculation means 134 adds the similarities calculated by the combination similarity calculation means 133 in the time domain, and calculates the timbre similarity (an example of the first similarity) of the input sound and the reference sound. The rhythm similarity calculation means 135 calculates the similarity between the observation activation matrix and the reference activation matrix. This similarity indicates the rhythm similarity (an example of the second similarity) of the input sound and the reference sound.

  FIG. 5 is a diagram illustrating a detailed functional configuration of the similarity calculation unit 15. The similarity calculation means 15 outputs the rhythm similarity calculated by an algorithm different from that of the similarity calculation means 13 for the input sound signal. The similarity calculation unit 15 includes a BPM acquisition unit 151, a normalization unit 152, a BS calculation unit 153, a reference BS acquisition unit 154, and a similarity calculation unit 155. The BPM acquisition unit 151 acquires BPM (Beat Per Minute) of the input sound signal, that is, the number of beats per unit time. The normalizing means 152 normalizes the input sound signal with BPM. Here, “normalize the input sound signal with BPM” means not only that the input sound signal is directly normalized with BPM, but also the signal obtained by applying some signal processing to the input sound signal with BPM. Including normalization. BS calculation means 153 (an example of first calculation means) calculates the beat spectrum of the normalized input sound signal. The reference BS acquisition unit 154 acquires a normalized beat spectrum obtained from the reference sound signal. The similarity calculation means 155 (an example of the second calculation means) compares the normalized beat spectrum of the input sound signal with the normalized beat spectrum of the reference sound signal, and compares the rhythm similarity between the input sound and the reference sound. Calculate

  FIG. 6 is a diagram illustrating a hardware configuration of the electronic musical instrument 10. The electronic musical instrument 10 includes a performance operator 101, a sound source 102, a sound generation control unit 103, an output unit 104, a storage 105, a CPU 106, and a communication IF 107. The performance operator 101 is an operator for a user (performer) to perform a performance operation, for example, a keyboard for a keyboard instrument, a string for a stringed instrument, or a key for a wind instrument. The sound source 102 stores sound data corresponding to each performance operator. For example, in a keyboard instrument, sound data corresponding to a certain key is data indicating a sound waveform from the rising edge to the disappearing edge of a sound generated when the key is pressed. The sound generation control unit 103 reads sound data from the sound source 102 in accordance with the operation of the performance operator 101. The output unit 104 outputs a sound signal corresponding to the read data (hereinafter referred to as “performance sound signal”). The storage 105 is a non-volatile storage device that stores data. The data stored in the storage 105 includes a database that records a plurality of accompaniment data. The CPU 106 is a control device that controls each unit of the electronic musical instrument 10. The CPU 106 supplies the accompaniment data read from the storage 105 to the output unit 104. The output unit 104 is an output device that outputs a sound signal corresponding to accompaniment data (hereinafter referred to as “accompaniment sound signal”) in addition to the performance sound signal, and includes, for example, a speaker. The communication IF 107 is an interface for communicating with other apparatuses, in this example, the information processing apparatus 20 in particular. The communication IF 107 communicates with the information processing apparatus 20 by wireless communication according to a predetermined standard, for example.

  FIG. 7 is a diagram illustrating a hardware configuration of the information processing apparatus 20. The information processing device 20 is a computer device that functions as a user terminal, for example, a smartphone. The information processing apparatus 20 includes a CPU 201, a memory 202, a storage 203, an input unit 204, an output unit 205, and a communication IF 206. The CPU 201 is a control device that controls other components of the information processing apparatus 20. The memory 202 is a volatile storage device that functions as a work space when the CPU 201 executes a program. The storage 203 is a non-volatile storage device that stores various data and programs. The input unit 204 is an input device that receives an instruction or information input from a user, and includes, for example, at least one of a touch sensor, a button, and a microphone. The output unit 205 is an output device that outputs information to the outside, and includes, for example, at least one of a display and a speaker. The communication IF 206 is an interface for communicating with other devices such as the electronic musical instrument 10 or a server device (not shown) on the network.

  In this example, among the functions of the music search system 1 shown in FIG. 2, the acquisition unit 11, identification unit 12, similarity calculation unit 13, database 14, similarity calculation unit 15, integration unit 16, and selection unit 17 are It is mounted on the information processing apparatus 20. The output means 18 is mounted on the electronic musical instrument 10.

  In the information processing apparatus 20, a program for causing a computer apparatus to function as a user terminal in the music search system 1 is stored in the storage 203. When the CPU 201 executes this program, the information processing device 20 functions as the acquisition unit 11, the identification unit 12, the similarity calculation unit 13, the database 14, the similarity calculation unit 15, the integration unit 16, and the selection unit 17. Implemented. The CPU 201 executing this program is an example of the acquisition unit 11, the identification unit 12, the similarity calculation unit 13, the similarity calculation unit 15, the integration unit 16, and the selection unit 17. The storage 203 is an example of the database 14. In the electronic musical instrument 10, the output unit 104 is an example of the output unit 18.

2. Operation 2-1. Outline FIG. 8 is a flowchart showing an outline of the operation of the music search system 1. The flow in FIG. 8 is started when the user inputs an instruction to start searching for music, for example. In step S1, the acquisition unit 11 acquires an input sound signal. In step S2, the specifying unit 12 performs a target section specifying process. In step S3, the similarity calculation means 13 performs similarity calculation by NMF. In step S4, the similarity calculation means 15 performs similarity calculation using a beat spectrum. In step S5, the integration unit 16 integrates the similarity by NMF and the similarity by beat spectrum. In step S6, the selection means 17 selects a music piece based on the integrated similarity. In step S7, the output means 18 outputs the selected music piece. That is, the output means 18 outputs an accompaniment sound similar to the input sound. Details of each process will be described below.

2-2. Target Section Identification Processing The similarity calculation in steps S3 and S4 may be performed on the entire input sound signal. However, when the entire input sound signal is targeted, there are the following problems. First, if the entire input sound signal is targeted, the time required for the calculation is required. Secondly, there are cases where the so-called intro or outro (ending) of the input sound signal includes a portion having no rhythm, and if similarity is calculated including such a portion, the reliability of similarity is calculated. Is that it will decrease. In the present embodiment, in order to cope with this point, only a part of the input sound signal that is the target of similarity calculation is limited.

  FIG. 9 is a flowchart showing details of the target section specifying process in step S2. In step S21, the specifying unit 12 performs a music structure analysis on the input sound signal. The music structure analysis is a process of analyzing a musical structure (so-called intro, A melody, B melody, chorus, and outro (ending)).

  FIG. 10 is a flowchart illustrating details of music structure analysis. In step S211, the specifying unit 12 divides the input sound signal into a plurality of unit sections. The unit section is, for example, a section corresponding to one measure of music. The division of the unit section is performed as follows, for example. First, the specifying unit 12 detects a beat point in the input sound signal. Next, the specifying unit 12 defines a section composed of a plurality of beat points corresponding to one measure as a unit section. For example, a technique described in Japanese Patent Application Laid-Open No. 2015-114361 is used to detect beat points and to define a section corresponding to one measure.

  In step S212, the specifying unit 12 calculates a timbre feature amount (hereinafter referred to as “tone color feature amount”) from the input sound signal. For example, a predetermined number (for example, 12) of MFCC (Mel-Frequency Cepstrum Coefficient) is used as the timbre feature amount. The MFCC is calculated for each unit section defined in step S211.

  In step S <b> 213, the specifying unit 12 calculates a chord feature amount (hereinafter referred to as “chord feature amount”) from the input sound signal. The chord feature amount is calculated for each frame (for example, a period corresponding to an eighth note or a sixteenth note) in which a unit section is further subdivided based on beat points. As the chord feature value, for example, a so-called chroma vector is used. The chroma vector is obtained by dividing the energy in the frequency range obtained by spectrum analysis into, for example, semitones and adding them within one octave. When divided into semitones, there are 12 sounds in total, so the chroma vector is a 12-dimensional vector. The chroma vector calculated for each frame represents the time change of the chord, that is, the chord progression.

  In step S214, the specifying unit 12 estimates the music structure of the input sound by posterior distribution estimation using a probability model. That is, the specifying means 12 uses a probability model describing the probability that a time series of feature values is observed under a certain music structure, and calculates the posterior probability when the time series of timbre feature values and chord feature values are observed. Estimate probability distribution (posterior distribution).

  As the probability model, for example, a music structure model, a timbre observation model, and a chord observation model are used. The music structure model is a model in which the music structure is described stochastically. The timbre observation model is a model that stochastically describes the generation process of the timbre feature. The chord observation model is a model that stochastically describes the generation process of chord features. In these probability models, each unit section is grouped into the same structure section with similar or common musical structures. Each group is distinguished by a section code (for example, A, B, C,...).

  The music structure model is, for example, a state transition model in which a plurality of states linked to each other are arranged in a state space, more specifically, a hidden Markov model. The timbre observation model follows, for example, an infinite mixed Gaussian distribution that employs a normal distribution as a probability distribution, and is a probability model that depends on the section code but does not depend on the stay time in the structure section. The chord observation model follows, for example, an infinite mixed Gaussian distribution that employs a normal distribution as the probability distribution, and is a probability model that depends on both the section code and the stay time in the structure section. The posterior distribution in each probability model is estimated by an iterative estimation algorithm such as a variational Bayes method. The specifying unit 12 estimates a music structure that maximizes the posterior distribution.

  In step S215, the specifying unit 12 specifies the music structure based on the estimation result in step S214.

  FIG. 11 is a diagram illustrating the music structure specified for the input sound signal. In this example, the input sound signal is divided into nine unit sections (τ1 to τ9). To these unit sections, section codes A, B, C, C, C, D, B, E, and F are assigned in order from the top.

  Refer to FIG. 9 again. In step S22, the specifying unit 12 divides the input sound signal. Specifically, the specifying unit 12 divides the input sound signal into unit sections according to the result of the music structure analysis. In step S <b> 23, the specifying unit 12 selects a section (hereinafter referred to as “target section”) used for the subsequent processing from the input sound signal divided into a plurality of parts.

  FIG. 12 is a flowchart illustrating the details of the target section selection process. In step S231, the specifying unit 12 calculates the priority of each unit section. In this example, a high priority is given to a large number of unit sections to which the same section code is assigned, and a low priority is given to a small number of unit sections. For example, in the example of FIG. 11, since there are three sections to which the section code C is assigned, these have priority 3, and since there are two sections to which the section code B is assigned, these have priority 2. Priorities 1 are assigned to the other sections. That is, in step S23, a section that is a target of rhythm similarity calculation is selected from a plurality of unit sections in descending order of sections classified into the same group in the music structure analysis.

  In addition, the reference | standard which gives a priority is not limited to said example. Other criteria may be used instead of or in addition to the above examples. As an example, for example, a standard that gives a high priority to a unit section having a long time length and gives a low priority to a unit section having a short time length is used. That is, step S23 in this other example is to select a section that is a target of calculation of the rhythm similarity in descending order of time length among a plurality of unit sections. In the example of FIG. 11, the time lengths of all the unit sections are the same. However, for example, when the tempo changes in the middle of a song, or when an algorithm in which a plurality of continuous unit sections are integrated in music structure analysis is adopted, Criteria that give priority based on length are meaningful. As another example, for example, according to the position on the time axis in the input sound signal, for example, a low priority is given to a section from the start to a predetermined time, and from a predetermined time before the end to the end, and other sections A criterion may be used that gives a higher priority to. These criteria may be weighted and added together.

  In step S232, the specifying unit 12 adds, to the target section, a section having the highest priority among sections not yet selected as the target section (hereinafter referred to as “non-selected section”). When there are a plurality of sections with the highest priority, the specifying unit 12 adds one section selected according to another criterion, for example, the section with the earliest number, to the target section.

  In step S233, the specifying unit 12 determines whether the cumulative time length of the target section has exceeded a threshold value. As the threshold value, for example, a predetermined ratio with respect to the total time length of the input sound signal, for example, 50% is used. When it is determined that the accumulated time length of the target section does not exceed the threshold value (S233: NO), the specifying unit 12 moves the process to step S232. When it is determined that the accumulated time length of the target section exceeds the threshold value (S233: YES), the specifying unit 12 ends the flow of FIG.

  In the example of FIG. 11, the section τ3 is first added to the target section, and thereafter, every time the process is repeated, the sections τ4, τ5, τ2, and τ7 are added to the target section in this order. In this example, since the time lengths of the sections τ1 to τ9 are equal, when the section τ7 is added to the target section, the target section becomes five sections in total, and the cumulative time length of the target section is 50% of the total time length of the input sound signal. Over.

  Refer to FIG. 9 again. In step S24, the specifying unit 12 specifies a target section based on the result of step S23. In the example of FIG. 11, sections τ1, τ4, τ5, τ2, and τ7 are specified as target sections. The specifying unit 12 generates a signal obtained by connecting only the target sections of the divided input sound signals. In the subsequent processing, this signal is processed as an input sound signal.

  According to this example, a part selected based on the musical structure of the input sound signal, for example, a section that repeatedly appears can be limited as a target of subsequent processing. Such a section is often a musically strong part such as so-called rust and A melody. By excluding from the processing target parts such as intro and outro that the rhythm and timbre may be different from other parts, it is possible to reduce the processing load while maintaining the accuracy of the search.

ここでＹは与えられた行列すなわち観測行列（ｍ行ｎ列）を示す。Ｈを基底行列（ｍ行ｋ列）といい、Ｕをアクティベーション（または係数）行列（ｋ行ｎ列）という。すなわちＮＭＦは、観測行列Ｙを、基底行列Ｈとアクティベーション行列Ｕとの積で近似する処理である。 2-3. Similarity Calculation by NMF Next, the similarity calculation by NMF in step S3 will be described. Before describing the details of the similarity calculation, an overview of NMF will be described first. NMF is a low rank approximation algorithm that decomposes a non-negative matrix into a product of two non-negative matrices. A non-negative matrix means a matrix whose components are all non-negative values (that is, zero or positive values). In general, NMF is represented by the following formula (1).

Here, Y represents a given matrix, that is, an observation matrix (m rows and n columns). H is referred to as a basis matrix (m rows and k columns), and U is referred to as an activation (or coefficient) matrix (k rows and n columns). That is, NMF is a process of approximating the observation matrix Y by the product of the base matrix H and the activation matrix U.

  In order to apply NMF to music similarity calculation, it is considered to use a matrix representing an amplitude spectrogram of a sound signal as the observation matrix Y. The amplitude spectrogram represents the time change of the frequency spectrum of the sound signal, and is three-dimensional information including time, frequency, and amplitude. The amplitude spectrogram is obtained, for example, by taking an absolute value with respect to a complex spectrogram obtained by sampling a sound signal in the time domain and subjecting the sound signal to a short-time Fourier transform. Here, when the horizontal axis is divided into n and the vertical axis is divided into m, and the amplitude in each divided region is digitized, the amplitude spectrogram can be represented as a matrix. This matrix includes temporal information in the row direction, frequency information in the column direction, and each component value includes information related to amplitude. Since the amplitude value is non-negative, this matrix is a non-negative matrix.

  FIG. 13 is a diagram showing an outline of NMF for an amplitude spectrogram. The example which applied NMF to the observation matrix Y obtained from the amplitude spectrogram is shown. The base matrix H includes a component related to frequency (an example of a first component) and a component related to a time (an example of a second component), and represents a set of representative spectral patterns included in an amplitude spectrogram. The activation matrix U can be considered to represent “at what timing” and “how strong” the representative spectrum pattern appears. More specifically, the basis matrix H includes a plurality of (two in the example of FIG. 13) basis vectors h corresponding to different sound sources. Each basis vector represents a typical frequency spectrum of a certain sound source. For example, the basis vector h (1) represents a typical spectrum pattern of the flute, and the basis vector h (2) represents a typical spectrum pattern of the clarinet. The activation matrix U includes a plurality (two in the example of FIG. 13) of activation vectors u corresponding to each sound source. For example, the activation vector u (1) represents the timing and intensity of the appearance of the flute spectrum pattern, and the activation vector u (2) represents the timing and intensity of the appearance of the clarinet spectrum pattern ( In the example of FIG. 13, the value of the component of the activation vector u is a binary value of on or off for the sake of simplicity.

なお、上式は、距離Ｄを最小にする行列ＨおよびＵを計算することを意味する。以降の式についても同様である。 The NMF calculates the base matrix H and the activation matrix U when the observation matrix Y is known. Specifically, NMF is defined as a problem that minimizes the distance D between the matrix Y and the matrix product HU, as in the following equation (2). As the distance D, for example, Euclidean distance, generalized KL distance, Itakura Saito distance, or β divergence is used. Although the solution of equation (2) cannot be obtained in closed form, several efficient iterative solutions are known (eg, Lee DD, & Sueng, HS (2001), Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13 (1) V621-V624).

The above equation means that the matrices H and U that minimize the distance D are calculated. The same applies to the following equations.

  If the musical instrument included in the input sound and the accompaniment sound is known to some extent, that is, if the musical instrument candidates included in the input sound and the accompaniment sound are limited to some extent in advance, the semi-supervised NMF is applied. May be. Semi-supervised NMF is described, for example, in Smaragdis P, Raj B, Shashanka MV. Supervised and Semi-supervised Separation of Sounds from Single-Channel Mixtures, In: ICA. 2007. p. 414-421.

  FIG. 14 is a flowchart illustrating details of similarity calculation by NMF. In step S31, the similarity calculation unit 13 calculates an amplitude spectrogram of the input sound signal. In step S32, the similarity calculation unit 13 applies NMF to the amplitude spectrogram of the input sound signal. Specifically, the similarity calculation means 13 first obtains an observation matrix Yo by matrixing the amplitude spectrogram of the input sound signal. Next, the similarity calculation means 13 applies NMF to the observation matrix Yo to calculate the observation base matrix Ho (an example of the first matrix) and the observation activation matrix Uo. That is, step S32 calculates the first matrix by a predetermined algorithm.

  In step S33, the similarity calculation unit 13 acquires a reference basis matrix Hr (an example of a second matrix) and a reference activation matrix Ur of the reference sound signal. In this example, NMF is applied in advance to each of a plurality of accompaniment data, and a reference basis matrix and a reference activation matrix are calculated. The calculated reference basis matrix and reference activation matrix are recorded in the database 14 as information on accompaniment data. The similarity calculation means 13 sequentially selects an accompaniment sound as a reference sound from a plurality of accompaniment data recorded in the database, and obtains a reference basis matrix and a reference activation matrix corresponding to the accompaniment sound. 14 from.

  Note that the reference basis matrix and the reference activation matrix recorded in the database 14 do not necessarily have to be calculated using the entire reference sound. The reference base matrix and the reference activation matrix may be calculated by applying NMF to only a part of the sections specified by the same process as the target section specifying process for the input sound.

  In step S34, the similarity calculation means 13 calculates the combinational similarity of the bases in each frame. The combination of bases refers to a combination of base vectors activated in a certain period among a plurality of base vectors included in the base matrix.

  FIG. 15 is a diagram illustrating combinations of bases. FIG. 15A schematically shows the NMF result corresponding to the input sound, and FIG. 15B schematically shows the NMF result corresponding to the reference sound. In this example, the basis matrices corresponding to the input sound and the reference sound all include basis vectors corresponding to the guitar, bass, hi-hat, snare, and bass drum. In the figure, activation vectors corresponding to each base vector are schematically shown. The horizontal axis represents time, and the vertical axis represents activation intensity. Looking at the combination of bases, for example, in the frame F1, the guitar, bass, hi-hat, and bass drum are activated in the input sound, and the hi-hat and bass drum are activated in the reference sound.

  The combination similarity of the base is obtained, for example, by extracting a column vector corresponding to a certain frame from the activation matrix for each of the input sound and the reference sound and calculating the inner product of both. This inner product indicates the combinational similarity of the basis in one frame. That is, step S34 calculates the similarity of the combination of the first components in the first matrix and the second matrix for each second component.

  Refer to FIG. 14 again. In step S <b> 35, the similarity calculation unit 13 calculates the timbre similarity between the input sound and the reference sound by integrating the combination similarity of each frame. That is, in step S35, the similarity of the combination of the first components is integrated for the second component to obtain the first similarity regarding the tone color of the input sound signal and the reference sound signal.

  Refer to FIG. 14 again. In step S36, the similarity calculation unit 13 calculates the rhythm similarity. In this example, the similarity of the activation vector corresponding to a specific base vector is used as the rhythm similarity. The specific basis vector is a basis vector corresponding to the musical instrument associated with the rhythm. That is, step S36 calculates the similarity of the time change in a specific 1st component among the 1st matrix and the 2nd matrix, and obtains the 2nd similarity concerning the rhythm of an input sound signal and a reference sound signal. Step S36 is an example of a step of calculating a rhythm similarity with the reference sound signal for at least a part of a plurality of sections included in the input sound signal. In the example of FIG. 15, the similarity of the activation vector corresponding to the bass drum is calculated. The processes of steps S33 to S36 are repeated until the timbre similarity and rhythm similarity are finally calculated for all accompaniment data while sequentially updating the reference sound.

  According to this example, not only the rhythm similarity but also the timbre similarity is calculated. Therefore, it is possible to search for music with higher accuracy than when only the rhythm similarity is used.

2-4. FIG. 16 is a flowchart illustrating the details of the similarity calculation based on the beat spectrum in step S4. The beat spectrum is a feature amount that captures a repetitive pattern on the spectrum, and is calculated by time-domain autocorrelation of some spectrogram-like feature amount. In this example, it is calculated by the autocorrelation of the spectral difference.

  In step S41, the similarity calculation means 15 acquires the BPM of the input sound signal. In this example, the similarity calculation means 15 calculates BPM by analyzing the input sound signal. A known method is used for calculating the BPM. In step S42, the similarity calculation unit 15 calculates an amplitude spectrogram of the input sound signal. In step S43, the similarity calculation unit 15 obtains a feature amount, in this example, a spectral difference, from the amplitude spectrogram. The spectral difference is a value obtained by taking the amplitude difference between adjacent frames on the time axis from the amplitude spectrogram. That is, the spectral difference is data of time difference on the horizontal axis and amplitude difference from the previous frame on the vertical axis. In step S44, the similarity calculation means 15 normalizes the input sound signal with the number of beats per unit time. Specifically, the similarity calculation means 15 normalizes the time axis of the spectrum difference with BPM. More specifically, the similarity calculation means 15 can normalize the time axis in units of 1 / n beats by dividing the time axis of the spectral difference by n times BPM.

  In step S45, the similarity calculation unit 15 calculates the beat spectrum of the normalized input sound signal. Specifically, the similarity calculation means 15 calculates a beat spectrum from the autocorrelation of normalized spectrum differences. In step S46, the similarity calculation unit 15 acquires a normalized beat spectrum of the reference sound signal. In this example, a beat spectrum is calculated in advance for each of a plurality of accompaniment data. The calculated beat spectrum is recorded in the database 14 as information on accompaniment data. The similarity calculation means 13 sequentially selects an accompaniment sound as a reference sound from a plurality of accompaniment data recorded in the database, and acquires a beat spectrum corresponding to the accompaniment sound from the database 14. In step S47, the similarity calculation means 15 compares the beat spectrum of the normalized input sound signal with the normalized beat spectrum calculated from the reference sound signal, and calculates the rhythm similarity. Specifically, the similarity calculation means 15 calculates the similarity between the beat spectra of the input sound and the accompaniment sound. Step S47 is another example of the step of calculating the rhythm similarity with the reference sound signal for at least a part of the plurality of sections included in the input sound signal.

  FIG. 17 is a diagram illustrating a beat spectrum. FIG. 17A shows the beat spectrum of the input sound, and FIG. 17B shows the beat spectrum of the reference sound. In the figure, the horizontal axis indicates the normalized beat frequency, and the vertical axis indicates the spectrum intensity. The similarity calculation means 15 calculates the similarity between the two spectra by pattern matching. Specifically, the beat spectrum is characterized by the frequency at which the peak appears and its peak intensity. The similarity calculation means 15 quantifies the beat spectrum by, for example, extracting the peak frequency and peak intensity as a feature amount for a peak having a peak intensity equal to or greater than a threshold value. The similarity calculation means 15 calculates the similarity between both using these feature quantities. This similarity is a rhythm similarity (an example of a fourth similarity). That is, step S47 calculates the similarity between the beat spectrum of the input sound signal and the beat spectrum of the reference sound signal to obtain the fourth similarity related to the rhythm.

  In similarity calculation using NMF, rhythm similarity is calculated from an activation matrix. However, in general, NMF has insufficient time resolution, and it is impossible to determine a difference in so-called even rhythm structure such as even or shuffle. Although it is possible to calculate by dividing the time more finely in NMF, there is a problem that the amount of calculation increases remarkably. Further, although the example of FIG. 15 shows an example in which the bases of the instruments are cleanly separated, as a general problem with NMF, the decomposition of instrument sounds does not always work well. Therefore, when instrument sounds cannot be separated well, there is a problem that the rhythm structure cannot be accurately captured by NMF.

  On the other hand, in this example, the rhythm similarity is calculated using the beat spectrum. Therefore, a fine rhythm structure can be grasped more accurately. Further, in the beat spectrum, the difference in BPM generally affects the feature quantity, and therefore it is difficult to evaluate the rhythm structure as the rhythm similarity even if the beat spectra are simply compared. However, in this example, the spectrum difference is normalized by BPM before the beat spectrum is calculated, and the difference in BPM in the input sound and the reference sound is absorbed.

2-5. Integration of similarity and selection of music The integration of similarity in step S5 is performed in detail as follows. In this example, two similarities (tone color similarity and rhythm similarity) are obtained by NMF, and one similarity (rhythm similarity) is obtained by the beat spectrum. These similarities are normalized to a common scale (for example, the lowest similarity is zero and the highest similarity is 1).

The integration unit 16 integrates a plurality of similarities by a weighting operation adjusted so that the similarity by NMF and the similarity by beat spectrum are predetermined weights, in this example, 1: 1. Specifically, the integration unit 16 calculates the similarity Di (an example of the third similarity) integrated by the following equation (3).
Di = 2 · DtN + DrN + Drb (3)
Here, DtN and DrN indicate the timbre similarity and rhythm similarity obtained by NMF, and Drb indicates the rhythm similarity obtained by the beat spectrum. According to this example, the similarity based on the NMF and the similarity based on the beat spectrum are evaluated with the same weight. The integrated similarity is calculated for each of the plurality of accompaniment data.

  The selection means 17 selects the accompaniment data having the highest similarity to the input sound from among the plurality of accompaniment data. In this example, since the selection means 17 is in the information processing apparatus 20 and the output means 18 is in the electronic musical instrument 10, the information processing apparatus 20 notifies the electronic musical instrument 10 of the identifier of the accompaniment data selected by the selection means 17. . In the electronic musical instrument 10, the output means 18 reads accompaniment data corresponding to the notified identifier, and outputs an accompaniment sound, that is, a music piece.

3. Modifications The present invention is not limited to the above-described embodiments, and various modifications can be made. Hereinafter, some modifications will be described. Two or more of the following modifications may be used in combination.

  The correspondence relationship between the functional configuration and the hardware configuration in the music search system 1 is not limited to the example described in the embodiment. For example, the music search system 1 may be one in which all functions are integrated in the information processing apparatus 20. In this case, the music to be searched is not limited to the accompaniment sound in the electronic musical instrument. For example, the music search system 1 may be applied to search for general music content played on a music player. Or the music search system 1 may be applied to the search of the music in a karaoke apparatus. Moreover, you may mount a part of function of the information processing apparatus 20 in the server apparatus on a network. For example, among the functions of the music search system 1, the specifying unit 12, the similarity calculating unit 13, the database 14, the similarity calculating unit 15, the integrating unit 16, and the selecting unit 17 may be mounted on the server device. In this case, when acquiring the input sound signal, the information processing device 20 transmits a search request including the input sound signal converted into data to the server device. The server device searches for music similar to the input sound signal included in the received search request, and returns the result to the information processing device 20.

  The method by which the specifying unit 12 specifies the target section from the input sound signal is not limited to the example described in the embodiment. The specifying unit 12 may specify a selected section as a target section from among a plurality of sections obtained by music structure analysis, for example, randomly or in response to a user instruction. Further, the specifying unit 12 is not limited to performing the selection of the target section until the accumulated time length of the target section exceeds the threshold value. For example, the specifying unit 12 may select the target section until the number of sections selected as the target section exceeds a threshold value. Alternatively, the specifying unit 12 may select the target section until there is no section whose priority is higher than the threshold value.

  The signal processing performed on the target section specified by the specifying unit 12 is not limited to that performed by the similarity calculating unit 13 and the similarity calculating unit 15. Processing other than the similarity calculation may be performed on the target section specified by the specifying unit 12.

  The similarity calculation means 13 is not limited to the one that calculates both the rhythm similarity and the timbre similarity. The similarity calculation means 13 may calculate only one of the rhythm similarity and the timbre similarity. In the similarity calculation means 13, the reference matrix acquisition means 132 does not acquire the base matrix and activation matrix corresponding to the reference sound signal from the database 14, but acquires the reference sound signal itself from the database 14. The basis matrix and activation matrix may be calculated by

  Either one of the similarity calculation means 13 and the similarity calculation means 15 may be omitted. In this case, the integration unit 16 is not necessary, and the selection unit 17 selects a music piece based only on the similarity by either the similarity calculation unit 13 or the similarity calculation unit 15.

  The acquisition unit 11, the identification unit 12, the similarity calculation unit 13, the similarity calculation unit 15, the integration unit 16, and the selection unit 17 are not limited to those implemented in a computer device by software. At least a part of these may be implemented as hardware by a dedicated integrated circuit, for example.

  The program executed by the CPU 201 of the information processing apparatus 20 may be provided by a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or may be downloaded via a communication line such as the Internet. Also, this program need not comprise all the steps of FIG. For example, this program may include only step S1, step S2, and step S3. Further, this program may include only step S1, step S2, and step S4. Furthermore, this program may include only step S1 and step S4.

DESCRIPTION OF SYMBOLS 1 ... Music search system, 10 ... Electronic musical instrument, 11 ... Acquisition means, 12 ... Identification means, 13 ... Similarity calculation means, 14 ... Database, 15 ... Similarity calculation means, 16 ... Integration means, 17 ... Selection means, 18 DESCRIPTION OF SYMBOLS ... Output means, 20 ... Information processing apparatus, 101 ... Performance operator, 102 ... Sound source, 103 ... Sound generation control section, 104 ... Output section, 105 ... Storage, 106 ... CPU, 107 ... Communication IF, 121 ... Structure analysis means, DESCRIPTION OF SYMBOLS 122 ... Dividing means, 123 ... Selection means, 124 ... Signal generation means, 131 ... Observation matrix calculation means, 132 ... Reference matrix acquisition means, 133 ... Similarity calculation means, 134 ... Similarity calculation means, 135 ... Similarity calculation means 151 ... BPM acquisition means, 152 ... normalization means, 153 ... BS calculation means, 154 ... reference BS acquisition means, 155 ... similarity calculation means, 201 ... CPU, 20 ... storage, 203 ... communication IF, 204 ... input section, 105 ... Output section

Claims (5)

Obtaining an input sound signal;
Calculating a first matrix corresponding to an amplitude spectrogram of the input sound signal and including a first component related to frequency and a second component related to time by a predetermined algorithm;
Obtaining a second matrix corresponding to an amplitude spectrogram of a reference sound signal, including the first component and the second component, and calculated by the predetermined algorithm;
Calculating the similarity of the combination of the first components in the first matrix and the second matrix for each second component;
A sound signal processing method comprising: integrating the similarity of the combination of the first components with respect to the second component to obtain a first similarity related to a tone color of the input sound signal and the reference sound signal. Calculating a temporal change similarity in a specific first component of the first matrix and the second matrix to obtain a second similarity related to a rhythm of the input sound signal and the reference sound signal;
The sound signal processing method according to claim 1, further comprising: calculating a third similarity obtained by integrating the first similarity and the second similarity. Calculating a similarity between the beat spectrum of the input sound signal and the beat spectrum of the reference sound signal to obtain a fourth similarity related to the rhythm,
The sound signal processing method according to claim 2, wherein in the calculation of the third similarity, the first similarity, the second similarity, and the fourth similarity are integrated. In the calculation of the third similarity, the first similarity, the second similarity, and the weighting calculation adjusted so that the ratio between the similarity related to the timbre and the similarity related to the rhythm is 1: 1. The sound signal processing method according to claim 3, wherein the fourth similarity is integrated. First acquisition means for acquiring an input sound signal;
Observation matrix calculation means for calculating a first matrix corresponding to an amplitude spectrogram of the input sound signal and including a first component related to frequency and a second component related to time by a predetermined algorithm;
A reference matrix acquisition unit that corresponds to an amplitude spectrogram of a reference sound signal, includes the first component and the second component, and acquires a second matrix calculated by the predetermined algorithm;
Combination similarity calculation means for calculating the similarity of the combination of the first components in the first matrix and the second matrix for each second component;
A sound signal processing apparatus comprising: a timbre similarity calculating unit that integrates the similarity of the combination of the first components with respect to the second component and calculates a first similarity related to a timbre of the input sound signal and the reference sound signal.

Images