JP6263382B2 - Audio signal processing apparatus, audio signal processing apparatus control method, and program - Google Patents

Description

  The present invention relates to an audio signal processing device for extracting musical instrument sounds in music, a control method for the audio signal processing device, and a program.

  Conventionally, a method using an isolator (a kind of equalizer using a band division filter) is known as a method for extracting the sound of each instrument in a music piece. However, if the level of the mid range is lowered using an isolator, the bass drum, snare drum, and hi-hat sounds are simultaneously cut off in the case of percussion instruments. In other words, it is not possible to control by instrument such as the attenuation of only the snare drum or the attenuation of only the hi-hat.

  On the other hand, Patent Documents 1 to 3 are known as techniques for extracting musical instrument sounds from music. Patent Document 1 separates and extracts percussion instrument sounds such as bass drums and hi-hats and non-percussion instrument sounds by collating spectrograms with templates (profile spectra). Further, Patent Document 2 separates and extracts percussion instrument sounds and non-percussion instrument sounds by paying attention to the spectrogram frequency and anisotropy of components in the time direction without using a template or iterative estimation. Further, Patent Document 3 separates instrument sounds from mixed sounds, and generates separated sounds using a sound of a sound module that outputs a specific sound simulating a specific instrument sound as a template.

Special table 2007-536587 gazette JP 2009-210888 A JP2011-209592A

  However, the techniques of Patent Document 1 and Patent Document 3 require a template as prior information. Moreover, although the technique of patent document 2 can isolate | separate percussion instrument sound and non-instrument sound, it cannot estimate the spectrogram shape for every percussion instrument.

  In view of the above problems, an object of the present invention is to provide an audio signal processing method, an audio signal processing apparatus, and a program that can estimate a spectrogram shape for each percussion instrument without using a template.

  The audio signal processing device according to the present invention is based on a first specifying unit that specifies a first spectrogram that is a frequency spectrogram of an arbitrary instrument from a predetermined sounding section, and a correlation value between the predetermined sounding section and the first spectrogram. And a second specifying unit for specifying a second spectrogram, which is a frequency spectrogram of the same instrument as an arbitrary instrument, and a synchronous subtracting unit for extracting a common component of the first spectrogram and the second spectrogram. To do.

  In the audio signal processing apparatus, the second specifying unit, when there are a plurality of frequency spectrograms of the same instrument as an arbitrary instrument in a predetermined tone generation section, obtains a frequency spectrogram having the highest correlation value with the first spectrogram. It is characterized by specifying as.

  In the above audio signal processing apparatus, when there are L frequency spectrograms of an arbitrary musical instrument in a predetermined sound generation section (where L is an integer satisfying L ≧ 2), L pieces of sound extracted by the synchronous subtraction unit It further includes a synchronous adder that averages the common components and calculates a common spectrogram.

  In the above audio signal processing apparatus, by replacing the L frequency spectrograms of an arbitrary musical instrument existing in a predetermined sound generation section with the common spectrogram calculated by the synchronous addition unit, the synchronously processed sound source of the arbitrary musical instrument can be obtained. It further comprises a sound source generating unit for generating.

  The above audio signal processing apparatus further includes a sound source separation unit that separates a synchronization-processed sound source of an arbitrary instrument generated by the sound source generation unit from an arbitrary piece of music, and the arbitrary piece of music includes a plurality of instrument sounds. If it is, the instrument sound separation process including the processes of the first specifying unit, the second specifying unit, the synchronization subtracting unit, the synchronization adding unit, the sound source generating unit, and the sound source separating unit is executed for each instrument sound. .

  In the above audio signal processing apparatus, a plurality of musical instrument sounds are a bass drum, a snare drum, and a hi-hat, and there is a hi-hat that is sounding alone and a hi-hat that is sounding simultaneously with another percussion instrument in an arbitrary musical piece. In this case, the musical instrument sound separation process is characterized in that the synchronized sound source is separated in the order of a hi-hat sounded simultaneously with another percussion instrument, a bass drum, a snare drum, and a hi-hat sounded independently.

  In the above audio signal processing device, a synchronization subtracting unit further includes a velocity specifying unit for specifying a velocity for each percussion instrument for each unit time obtained by equally dividing a predetermined sound generation section for a frequency band defined for each percussion instrument Is characterized in that the common component is extracted after aligning the amplitude values of the first spectrogram and the second spectrogram based on the velocity for each percussion instrument.

  The audio signal processing device further includes a detailed determination unit that groups a plurality of frequency spectrograms existing in a predetermined sound generation section on condition that the same percussion instrument type and the same sounding method are used, and the second specifying unit Specifies a second spectrogram from one or more frequency spectrograms belonging to the same group as the first spectrogram.

  The audio signal processing apparatus further includes a sound generation position information acquisition unit that acquires sound position information indicating the sound generation positions of the three percussion instruments, and the detailed determination unit includes a first spectrogram in a predetermined sound generation section from the sound generation position information. When it is known that there are a plurality of frequency spectrograms of the same percussion instrument, the average value of the correlation values of the plurality of frequency spectrograms with respect to the first spectrogram is calculated, and the frequency spectrogram of the correlation value exceeding the average value is calculated as the first spectrogram. It is classified as the same group as the spectrogram.

  In the above audio signal processing device, based on the amplitude spectrum information obtained by performing frequency Fourier transform on the audio signal of an arbitrary piece of music, a continuous sound component continuing for a predetermined time or more is extracted and removed from the audio signal. A continuous sound removing unit is further provided, wherein the first specifying unit and the second specifying unit specify the first spectrogram and the second spectrogram after the continuous sound component is removed.

  The control method of the audio signal processing device according to the present invention includes a first specifying step of specifying a first spectrogram that is a frequency spectrogram of an arbitrary instrument from a predetermined sounding section, and a correlation between the first sounding section from the predetermined sounding section. Performing a second specifying step for specifying a second spectrogram, which is a frequency spectrogram of the same instrument as an arbitrary instrument based on the value, and a synchronous subtracting step for extracting a common component of the first spectrogram and the second spectrogram. It is characterized by.

  A program according to the present invention causes a computer to execute each step in the above-described method for controlling an audio signal processing device.

It is a block diagram which shows the function structure of the audio | voice signal processing apparatus which concerns on one Embodiment of this invention. It is a detailed block diagram of the sound processing part for detailed determination. It is a detailed block diagram of a velocity specific part. It is a detailed block diagram of a percussion instrument sound separation unit. It is a flowchart which shows the flow of a continuous sound removal process. It is a flowchart which shows the flow of a minimum point detection process. It is a flowchart which shows the flow of local maximum detection processing. It is a flowchart which shows the flow of the protrusion degree determination process of a local maximum point. It is explanatory drawing of a continuation counter update process. It is a flowchart which shows the flow of a continuous sound range detection process. It is explanatory drawing of a continuous sound correction process. It is explanatory drawing of a bass drum processing process (sound end determination). It is explanatory drawing of a bass drum process (bass sound subtraction). It is explanatory drawing of a snare drum process. It is explanatory drawing of a hi-hat process. It is a flowchart which shows the flow of a detailed discrimination | determination process. It is explanatory drawing of a detailed discrimination | determination process. It is explanatory drawing of a grouping threshold value. It is a flowchart which shows the flow of a velocity specific process. It is explanatory drawing of a velocity detection process. It is explanatory drawing of a velocity calculation process. It is a flowchart which shows the flow of a percussion instrument sound separation process. It is explanatory drawing of a composite hi-hat separation process and a bass drum separation process. It is explanatory drawing of a synchronous subtraction process. It is explanatory drawing of a synchronous addition process. It is the simple flowchart which shows the flow to a re-attack detection process, and its explanatory drawing. It is explanatory drawing of a ringing end correction process. It is explanatory drawing regarding adjustment of a percussion instrument sound. It is explanatory drawing regarding the musical score display of a percussion instrument sound. It is a supplementary explanatory drawing of bass drum processing. It is explanatory drawing regarding the application example of a bass-drum processing process.

  Hereinafter, an audio signal processing device, a control method for the audio signal processing device, and a program according to an embodiment of the present invention will be described with reference to the accompanying drawings. The present invention estimates the spectrogram shape of a specific musical instrument in music and separates (extracts) the musical instrument sound. Therefore, in the present embodiment, the case where the spectrogram shapes of three types of percussion instruments (bass drum, snare drum, hi-hat) are estimated and these three percussion instrument sounds are illustrated is exemplified.

  FIG. 1 is a block diagram showing a functional configuration of an audio signal processing device 1 according to an embodiment of the present invention. The audio signal processing apparatus 1 includes, as main functional configurations, an FFT (Fast Fourier Transform) unit 11, a continuous sound removing unit 12, a band dividing unit 13, a detailed determination sound processing unit 14, a detailed determination unit 15, and a velocity specifying unit 16. A groove determination unit 17 and a percussion instrument sound separation unit 18. The audio signal processing device 1 may be a dedicated device, a DJ device (DJ player, DJ mixer, etc.), an audio device (CD player, DVD player, etc.), a sound editing device, a personal computer, a tablet terminal, It may be a part of various electronic devices such as an effector, a recording device, and a broadcasting device.

  The FFT unit 11 generates analysis data (amplitude spectrum information) by frequency Fourier transforming an input sound (sound signal of an arbitrary music such as a wav file). Here, the FFT size is 2048 samples and the number of overlaps is four. In this case, one frame (FFT processing interval) is 512 samples.

  The continuous sound removing unit 12 extracts a continuous sound component that has continued for a predetermined time or more based on the amplitude spectrum information obtained by frequency Fourier transform, and removes the continuous sound component from the input sound signal. By this processing, it is possible to remove “components that are not percussion instruments” that cause errors in specifying the spectrogram shape of each percussion instrument. Although not particularly illustrated, IFFT (Inverse FFT) is performed after the processing of the continuous sound removing unit 12 to temporarily return to the time axis, and then FFT is performed again. Specifically, the FFT size is 512 samples, and one frame is 128 samples. In this way, the sense of attack can be reproduced by changing the FFT size and then supplying data (continuous sound removal sound source) to the percussion instrument sound separation unit 18.

  The band dividing unit 13 divides the frequency band of each percussion instrument. For example, the frequency range corresponding to each percussion instrument is limited, such as a bass drum “40 to 300 Hz”, a snare drum “600 to 3000 Hz”, and a hi-hat “6000 Hz to 16000 Hz”.

  The detail determination sound processing unit 14 acquires sound generation position information from the outside, and performs processing on the input sound signal so as to be a component of only a percussion instrument. By this processing, the correct answer rate of the detail determining unit 15 at the latter stage can be increased. Note that the pronunciation position information is information generated based on the beat position analysis result of the music, and information indicating the sound generation position of each percussion instrument included in the music (where the bass drum, snare drum, and hi-hat sound) Information). In this embodiment, it is assumed that the analysis has already been performed in an external device (not shown).

  Based on the acquired sound generation position information, the detail determination unit 15 uses the same percussion instrument type and the same sounding method for a plurality of frequency spectrograms existing in a predetermined sound generation section (in this embodiment, eight measures of music). Grouping on the condition that By this process, the accuracy of the synchronization process in the percussion instrument sound separation unit 18 to be described later can be improved.

  In addition, when it is known from the acquired sound generation position information that there are a plurality of frequency spectrograms of the same percussion instrument as an arbitrary spectrogram (first spectrogram) from the acquired sound generation position information, the detailed determination unit 15 Calculate the average value of the correlation value of the spectrogram with respect to an arbitrary spectrogram, and classify the frequency spectrogram of the correlation value exceeding the average value into the same group as the arbitrary spectrogram (assuming the same percussion instrument type and the same sounding method) judge). Details will be described later.

  The velocity specifying unit 16 specifies velocity information of each percussion instrument in a predetermined sound generation section based on the acquired sound generation position information. This processing is also performed in order to increase the accuracy of the synchronization processing in the percussion instrument sound separation unit 18 described later.

  The groove determination unit 17 determines the groove of the music in the predetermined sounding section based on the velocity information of each percussion instrument in the predetermined sounding section specified by the velocity specifying unit 16 and the acquired sounding position information. The determination result is output as groove information. The velocity information refers to a velocity value for each unit time (for example, a sixteenth note unit) obtained by equally dividing one measure. Although details will be described later, the velocity for each unit time (unit time 1/16 to 16/16) is specified in common to all the bars included in the predetermined sound generation section (8 bars).

  The percussion instrument sound separation unit 18 uses the continuous sound removal sound source obtained by the above continuous sound removal unit 12 and is based on the detailed discrimination information (grouping information and velocity information) obtained by the detailed discrimination unit 15 and the velocity specifying unit 16. The spectrogram shape of each percussion instrument is estimated, and each percussion instrument sound is separated. Further, the spectrogram shape estimation result is output as spectrogram information (bass drum amplitude information, snare drum amplitude information, hi-hat amplitude information). In the drawings, the bass drum is abbreviated as “BD”, the snare drum as “SD”, and the hi-hat as “HH”.

  Next, with reference to FIG. 2 thru | or FIG. 4, the detailed functional structure of the sound processing part 14 for detailed determination, the velocity specific | specification part 16, and the percussion instrument sound separation part 18 is demonstrated. FIG. 2 is a detailed block diagram of the detail determination sound processing unit 14. The detail determination sound processing unit 14 includes a bass drum processing unit 21, a snare drum processing unit 22, and a hi-hat processing unit 23. The bass drum processing unit 21 performs processing for removing bass sounds and extracting bass drum sounds. The snare drum processing unit 22 eliminates human voices and piano accompaniment and performs processing for extracting the snare drum sound. The hi-hat processing unit 23 performs a process for eliminating other hi-hat sounds that are covered by the hi-hat sound to be extracted.

  Here, the bass drum processing unit 21 will be described in more detail. The bass drum processing unit 21 includes a pronunciation position information acquisition unit 21a, a search section identification unit 21b, a ringing end determination unit 21c, an extraction unit 21d, a first processing unit 21e, and a second processing unit 21f.

  The sound generation position information acquisition unit 21a acquires sound generation position information indicating the sound generation position of an arbitrary instrument (bass drum, snare drum, hi-hat) included in an arbitrary musical piece from the outside. The search section specifying unit 21b specifies a search section for searching for a bass drum sounding section based on the acquired sounding position information. In the present embodiment, a section consisting of a predetermined time before and after the attack position of the bass drum is specified as a search section.

  The ringing end determination unit 21c determines the end of ringing of the bass drum within the specified search section. In the present embodiment, the end of ringing is determined using two determination methods of “average value end point determination method” and “new attack determination method”. The former is a method of determining, as the end of sounding, a point in time when a moving average value for a plurality of frames continuously falls below a variation threshold value that is an average value near an attack position in a search section. The latter is a method in which when a sound different from the bass drum to be determined is generated, the time point when the other sound is generated is determined as the end of sounding.

  The extraction unit 21d extracts an amplitude value (hereinafter also referred to as “amplitude data”) at a predetermined position in the search section when the end of ringing is not determined by the end of ring determination unit 21c (when ringing has not ended). In the present embodiment, it is assumed that the amplitude value of the last frame in the search section is extracted.

  When the end of ringing is not determined by the end of ring determination unit 21c, the first processing unit 21e processes the voice data included in the search section based on the amplitude value extracted by the extraction unit 21d. In the present embodiment, the amplitude value extracted by the extraction unit 21d is subtracted from all frames included in the search section. Thereby, when the bass drum sound and the bass sound overlap, it is possible to exclude the bass sound and extract only the bass drum sound. On the other hand, when the end of ringing is determined by the ringing end determination unit 21c (when ringing is completed), the second processing unit 21f sets the amplitude value after the end of ringing in the search section to zero. Thereby, unnecessary sounds after the end of ringing can be eliminated.

  Although details will be described later, also in the snare drum processing unit 22 and the hi-hat processing unit 23, the pronunciation position information acquisition unit 21a, the search section specifying unit 21b, the extraction unit 21d, and the first processing unit 21e in the bass drum processing unit 21. The process is substantially the same as. As a modification, the snare drum processing unit 22 and the hi-hat processing unit 23 may include a squeal end determination unit 21c and a second processing unit 21f. That is, the end of ringing within the search section for the snare drum and hi-hat may be determined, and the amplitude value after the end of ringing may be set to zero.

  Next, FIG. 3 is a detailed block diagram of the velocity specifying unit 16. The velocity specifying unit 16 includes a velocity detecting unit 31 and a velocity calculating unit 32. Note that each part of the velocity specifying unit 16 is processed for each percussion instrument.

  The velocity detection unit 31 detects the velocity of each percussion instrument for a partial section within a predetermined sound generation section. For example, in the case of a bass drum, a section from the second sounding position obtained from the acquired sounding position information to the last sounding position among the eight measures of the music is targeted. In the case of a snare drum and a hi-hat, the section from the 3rd bar to the 6th bar among the 8 bars of the music is targeted. For any percussion instrument, the velocity of each percussion instrument is detected every unit time obtained by equally dividing each measure into N pieces (16 in this embodiment). Further, the velocity detection unit 31 has the largest amplitude intensity in a predetermined sounding section when the sum of the amplitude values in the sounding continuation period determined for each percussion instrument in the frequency range determined for each percussion instrument is defined as the amplitude intensity. The value normalized by is detected as the velocity of the corresponding percussion instrument.

  The velocity calculation unit 32 uses the detection result of the velocity detection unit 31 to calculate the velocity of each percussion instrument in a section excluding the partial section in a predetermined sound generation section. Specifically, the average value of each unit time from the first to the 16th in each measure included in the partial section is calculated as the velocity of each percussion instrument.

  Next, FIG. 4 is a detailed block diagram of the percussion instrument sound separation unit 18. The percussion instrument sound separation unit 18 includes a first identification unit 41, a second identification unit 42, a synchronization subtraction unit 43, a synchronization addition unit 44, a re-attack detection unit 45, a ringing end determination unit 46, a sound source generation unit 47, and a sound source separation unit 48. including. Each part of the percussion instrument sound separation unit 18 is also processed for each percussion instrument.

  The first specifying unit 41 specifies a first spectrogram that is a frequency spectrogram of an arbitrary percussion instrument from a predetermined sounding section after the continuous sound component is removed from the input sound signal by the continuous sound removing unit 12. Based on the correlation value with the first spectrogram from a predetermined sounding section, the second specifying unit 42 selects a frequency spectrogram of the same instrument as the first spectrogram from one or more frequency spectrograms belonging to the same group as an arbitrary percussion instrument. A second spectrogram that is Here, when there are a plurality of frequency spectrograms as identification candidates, the frequency spectrogram having the highest correlation value with the first spectrogram is identified.

  The synchronous subtraction unit 43 extracts a common component of the identified first spectrogram and second spectrogram. At this time, the synchronous subtraction unit 43 extracts the common component after aligning the amplitude values of the first spectrogram and the second spectrogram based on the velocity for each percussion instrument specified by the velocity specifying unit 16. When there are L frequency spectrograms of an arbitrary percussion instrument (where L is an integer satisfying L ≧ 2) in a predetermined sound generation section, the synchronous adder 44 uses the L commons extracted by the synchronous subtractor 43. The components are averaged to calculate a common spectrogram.

  The re-attack detection unit 45 performs attack detection using the synchronization processing results obtained by the synchronization subtraction unit 43 and the synchronization addition unit 44. With this processing, even when only the snare drum is pronounced forward relative to the bass drum or hi-hat, the attack position can be detected accurately. The ringing end determination unit 46 determines the end of ringing of an arbitrary percussion instrument and removes other components in the same frequency band as that of the arbitrary percussion instrument. By this process, components other than the percussion instrument that could not be removed by the synchronization process can be removed, and a sound like a percussion instrument can be processed.

  The sound source generation unit 47 replaces the L frequency spectrograms of each percussion instrument existing in a predetermined sounding section with the common spectrogram calculated by the synchronous addition unit 44, and the re-attack detection unit 45 and the end-of-sound determination unit 46 A synchronized sound source processed based on the processing result is generated. The sound source separation unit 48 separates the synchronized sound source of each percussion instrument generated by the sound source generation unit 47 from the input sound (arbitrary music). The instrument sound separation process executed by the percussion instrument sound separation unit 18 is sounded independently by a hi-hat (hereinafter referred to as “composite hi-hat”), bass drum, and snare drum that are sounded simultaneously with another percussion instrument. The operations are executed in the order of hi-hat (hereinafter referred to as “single hi-hat”).

  Next, with reference to FIG. 5 and subsequent figures, each part will be further described with a specific example. First, the continuous sound removal processing by the continuous sound removal unit 12 will be described with reference to FIGS. The continuous tone removal process is a process of removing “a component that is not a percussion instrument” as described above. FIG. 5 is a flowchart showing a flow of continuous sound removal processing by the audio signal processing apparatus 1.

  In the continuous sound removal process, the minimum point and the maximum point are detected from the amplitude spectrum information obtained by FFT (S11, S12), and the degree of protrusion of the maximum point is determined from these results (S13). Further, the continuation counter is updated based on the determination result (S14), and the continuation sound is determined (S15). Further, the continuous sound range is detected from the confirmed continuous sound (S16), and the erroneously detected continuous sound is corrected based on the detected continuous sound range (S17). Thereafter, the amplitude of the continuous sound is removed from the original sound (S18), and the continuous sound removal process is terminated.

  FIG. 6 is a flowchart showing the flow of the minimum point detection process (see S11 in FIG. 5). In the minimum point detection process, the frequency bin “0” is set as a start value, and the process is started for half of the FFT size (S21). First, an inclination with respect to the amplitude value of the bins adjacent to each other is obtained centering on the amplitude value of the target frequency bin (see the figure, reference P1) (S22). Here, when the target frequency bin is minimum (when the inclination with respect to both adjacent bins is greater than or equal to a predetermined value) (S23: Yes), the frequency bin is recorded as the minimum point frequency bin (S24). If the target frequency bin is not minimal (S23: No), S24 is omitted. Thereafter, S21 to S24 are repeated while sequentially increasing the target frequency bin (S25).

  FIG. 7 is a flowchart showing the flow of local maximum point detection processing (see S12 in FIG. 5). Even in the local maximum point detection process, the frequency bin “0” is set as a start value, and the process is started for half of the FFT size (S31). First, an inclination with respect to the amplitude value of the bins adjacent to each other is obtained centering on the amplitude value of the target frequency bin (see the figure, reference symbol P2) (S32). Here, when the target frequency bin is maximal (when the slope with respect to both adjacent bins is equal to or smaller than a predetermined value) (S33: Yes), it is recorded as the frequency bin of the maximal point (S34). If the target frequency bin is not maximal (S33: No), S34 is omitted. Thereafter, S31 to S34 are repeated while sequentially incrementing the target frequency bin (S35).

  FIG. 8 is a flowchart showing the flow of the maximum point protrusion degree determination process (see S13 in FIG. 5). This process is performed in order to eliminate the local maximum point due to the noise component. For example, when white noise or the like is input, innumerable small local maximum points may occur in the high frequency range. For this reason, such a noise component and a maximum point such as a voice or a musical instrument to be detected are distinguished from each other by a protrusion degree determination process of the maximum point, and a maximum point protruding to some extent from the amplitude value of the surrounding frequency is left.

  Also in the protrusion degree determination process, the frequency bin “0” is set as a start value, and the process is started for half of the FFT size (S41). First, a value (complementary value, the same figure) linearly complemented with the amplitude value (minimum value, the same figure, symbols P3 and P4) of the adjacent local minimum points for the frequency bin (see the symbol P5) of the target local maximum point. , (See symbol L11) (S42). When the linearly complemented value is large (when the complement value is greater than or equal to a predetermined value) (S43: Yes), it is recorded as the maximum point frequency bin (S44). When the linearly complemented value is small (S44: No), S44 is omitted. Thereafter, S41 to S44 are repeated while sequentially incrementing the target frequency bin (S45).

  FIG. 9 is an explanatory diagram of the continuation counter update process. In this process, a counter indicating how long the sound has continued is updated based on the maximum point recorded by the protrusion degree determination process. Specifically, if the local maximum point exists in the same frequency bin in the previous frame or exists in both adjacent frequency bins, it is determined that it is continuing and the counter is incremented. In the case of the example in the figure, the maximum point from the 0th frame to the 4th frame indicated by the arrow is counted as a continuous sound, and is determined to be a sound that has continued for 5 frames. Further, the local maximum point existing in the sixth frame (see P6 in the figure) is counted as a new sound because the continuation of the local maximum point is interrupted.

  FIG. 10 is a flowchart showing the flow of the continuous sound range detection process (see S16 in FIG. 5). This process is a process of counting the frequency bin having the highest frequency in each frame among the determined continuous sounds and detecting the continuous sound range based on the median value in the eight bars.

  In the continuous sound range detection process, the process is started with the frame “0” as a start value and a frame for 8 bars (S51). First, the frequency bin having the highest frequency is counted (S52), and it is determined whether or not a predetermined search range (for example, 0 Hz to 4000 Hz) is deviated (S53). If the search range is deviated (S53: Yes), the number of departures is recorded (S54). Thereafter, S51 to S54 are repeated while sequentially incrementing the target frequency frame (S55). Thereafter, the median value of the highest frequency that has been aggregated is calculated, and the range from 0 Hz to the median value is determined as the continuous sound range (S56). Further, if the number of times (time) deviating from the search range is more than half of the eight bars, it means that the sound is clogged, and a rust flag is set (S57). The rust flag is used for determining the end of a percussion instrument sound separation process, which will be described later.

  FIG. 11 is an explanatory diagram of the continuous sound correction process (see S17 of FIG. 5). This process corrects a continuation sound on the assumption that a continuation sound that is more than twice the continuous sound range is a false detection of a “percussion instrument but a continuous sound” such as an open hi-hat. . For example, as shown in FIG. 6A, a continuous sound component exceeding a frequency twice the determined continuous sound range is regarded as erroneous detection, and the erroneous detection property is removed as shown in FIG. .

  Next, the detailed discrimination sound processing by the detailed determination sound processing unit 14 will be described with reference to FIGS. As described above, the detailed discrimination sound processing is a process for generating a component of only a percussion instrument as a pre-processing of the detailed discrimination process. 12 and 13 are explanatory diagrams of the bass drum processing. In both figures, the symbol ta is the bass drum attack position (sound generation position), the symbol te is the bass drum ringing end position, the symbol t1 is the start position of the search section for searching for the percussion instrument ringing end, Symbol t2 indicates the end position of the search section. In the present embodiment, the start point and end point of the search section are defined as points in time before and after a predetermined time (for example, around several tens of ms to several hundred ms) from the attack position of each percussion instrument. Yes. The predetermined time before and after this time may be different for each percussion instrument.

  Since the detailed determination processing of the bass drum is performed only in the low range, the low-frequency amplitude information is processed. In the case of a bass drum, in order to remove the bass component as much as possible, processing that reflects the end of the bass drum sound is performed. FIG. 12A is an explanatory diagram of the average value end determination method. In this method, an average value of volume in a low frequency region near an attack position is calculated from an LPF (Low-pass filter) having a relatively large time constant, and is used as a variation threshold value. Then, a moving average for a plurality of frames (for example, 4 frames) is taken, and a point in time when the moving average value is continuously lower than the fluctuation threshold (see te in the figure) is determined as the end of ringing. The reason why the moving average for a plurality of frames is taken is that if the FFT size is small, the frequency resolution is low and the low-frequency waveform is disturbed (the disturbance can be reduced by increasing the FFT size). The reason why the determination is made using the variation threshold is to accurately detect the bass drum low frequency section without erroneously detecting the bass sound. Furthermore, if the waveform disturbance cannot be suppressed even if the waveform disturbance is suppressed by taking an average value for a plurality of frames, it may fall below the threshold for a moment. This is to avoid erroneous determination in such a case.

  On the other hand, FIG. 5B is an explanatory diagram of the new attack determination method. In this method, when any new sound is generated (see te in the same figure), it is determined that the ringing is finished. For example, as shown in the figure, there may be a case where the next bass drum is played before the bass drum is finished.

  When the end of ringing is determined by the average value end determination method of FIG. 12 (a) or the new attack determination method of FIG. 12 (b), as shown in FIG. The amplitude of (range) is set to “0”. In addition, as shown in FIG. 13A, when the end of ringing is not determined by either the average value end determination method or the new attack determination method in the search section (range t1 to t2), As shown in (b), the amplitude data (see reference numeral 51a in the figure) of the last frame in the search section is subtracted from the audio data included in the search section. FIG. 4C shows the subtraction result. Thus, when the bass sound is not attenuated, the influence of the bass sound can be eliminated by subtracting the amplitude data of the last frame in the search section.

  The section to be subtracted (the section that is the target of the first processing unit 21e) is not limited to the search section, and may be a section obtained by adding a predetermined time before and after the search section. Moreover, you may prescribe | regulate the area subtracted on the basis of an attack position irrespective of a search area. Further, the section to be subtracted when the end of ringing is determined (the section that is the target of the second processing unit 21f) may be a section that includes not only the end of the search section but also the end of the search section.

  FIG. 14 is an explanatory diagram of the snare drum processing. In the figure, reference symbol ta represents the attack position of the snare drum, and reference symbols t1 and t2 represent the start position and end position of the search section. Reference numeral t3 is a time point a predetermined time before the end position of the search section (a time point a predetermined time after the snare drum attack position), and indicates the extraction target position of the extraction unit 21d (see FIG. 2). .

  In the band of the snare drum, accompaniment such as voice and piano mainly affects the result of detailed discrimination. Therefore, amplitude data (see reference numeral 52 in the figure) before a predetermined time from the end position of the search section (see reference numeral 52 in the figure), that is, a time that is several tens of milliseconds from the snare drum attack position (see ta in the figure). By subtracting the amplitude data (see t3 in the figure), these effects are reduced. That is, when a sound other than the snare drum, such as a voice, is being sounded, as shown in FIG. 10A, the amplitude data of the frame at time t3 as shown in FIG. Is subtracted from all frames in the search interval. FIG. 4C shows the subtraction result. In this way, when sounds other than the snare drum continue to sound at the same pitch, the influence can be reduced by subtracting the amplitude data after a predetermined time has elapsed from the attack position.

  FIG. 15 is an explanatory diagram of the hi-hat processing. In the figure, reference symbol ta indicates the hi-hat attack position, and reference symbols t1 and t2 indicate the start position and end position of the search section. Reference sign t4 indicates a position to be extracted by the extraction unit 21d (see FIG. 2), which is a time point after a predetermined time from the start position of the search section (a time point before the hi-hat attack position). In the case of a hi-hat, the next hi-hat often sounds before the previous hi-hat finishes. In addition, unlike bass drums and snare drums, there is a high possibility of another type of hi-hat. For this reason, the sound generation is not forcibly stopped and a new sound is not sounded, but the previous hi-hat sound covers the newly generated hi-hat sound, which adversely affects detailed discrimination. To eliminate this, the past amplitude is subtracted. That is, when the hi-hat is sounding before the open hi-hat has finished ringing as shown in FIG. 10A, as shown in FIG. 10B, several tens of minutes from the hi-hat attack position (see ta in the same figure). Amplitude data (see reference numeral 53 in the figure) for the time obtained by ms (see t4 in the figure) is subtracted from all frames. FIG. 4C shows the subtraction result. As described above, even when the hi-hat is sounding before the open hi-hat is finished, the influence of the open hi-hat can be reduced by subtracting the amplitude data of a predetermined time from the hi-hat attack.

  In the snare drum processing shown in FIG. 14 and the hi-hat processing shown in FIG. 15, it is not always necessary to specify the search section. That is, in the snare drum processing unit 22 and the hi-hat processing unit 23, the search section specifying unit 21b in the bass drum processing unit 21 may be omitted. In this case, the interval to be subtracted may be defined as a predetermined time before and after the attack position.

  Next, with reference to FIGS. 16 to 18, the detailed determination processing by the detailed determination unit 15 will be described. As described above, the detailed discrimination process is performed in order to obtain information on “the same type (the hi-hat and the open hi-hat are different types), or the overlapping part of the same percussion instrument”, which is necessary in the percussion instrument sound separation process. Specifically, grouping is performed on condition that the same percussion instrument type and the same sounding method are used.

  FIG. 16 is a flowchart showing the flow of the detailed determination process by the audio signal processing apparatus 1. In the detailed discrimination process, discrimination is performed for each... (S61). Further, the attack “0” in the eight bars is set as an initial value, and processing is performed for all attacks (S62). First, it is determined whether or not the detected attack is the target percussion instrument (S63), and if it is the target percussion instrument (S63: Yes), a correlation value is calculated for the same type of percussion instrument (S64). As a result, attacks whose correlation value exceeds the threshold are regarded as the same group (S65). Thereafter, S62 to S65 are repeated while sequentially increasing the target attack (S66). Further, S61 to S66 are repeated for each percussion instrument (S67).

  FIG. 17 is an explanatory diagram of the detailed determination process. As shown in the figure, the correlation value of the frequency spectrogram between the same percussion instruments is used as a feature value for checking that the same percussion instrument type and the same sounding style are used. Here, since the same percussion instrument is originally collated based on the pronunciation position information, a high correlation value should be basically obtained. For example, when a plurality of frequency spectrograms are included in 8 bars as shown in FIG. 8A, an arbitrary frequency spectrogram from which correlation is obtained is determined (see FIG. Hi-hat in the example in the figure). Subsequently, as shown in FIG. 5B, the correlation value with an arbitrary frequency spectrogram is obtained for all the hi-hat frequency spectrograms included in the eight bars. As a result, as shown in FIG. 6C, the three frequency spectrograms except for the high hat (open hi-hat) having a low correlation value are determined to be the same group as the arbitrary frequency spectrogram (HH group 1 in the example in the figure). . A threshold (grouping threshold) used for grouping will be described later. Thereafter, the same processing is repeated for the frequency spectrograms of percussion instruments without group numbers, and the detailed discrimination processing is terminated when group numbers are assigned to all frequency spectrograms. The group number is output to the percussion instrument sound separation unit 18 as grouping information.

  FIG. 18 is an explanatory diagram of the grouping threshold. The grouping threshold value for determining that the same ringing is performed is obtained by a calculation formula “threshold = 1− (maximum correlation value−average correlation value) × 0.5”. That is, the median value of the maximum correlation value and the average correlation value is the grouping threshold. The example in the figure shows the calculation result of the correlation value with one original spectrogram among nine frequency spectrograms in eight bars for an arbitrary percussion instrument. Thus, by setting the grouping threshold as the variation threshold, more accurate grouping can be performed regardless of the music. The grouping threshold is determined for each percussion instrument.

  Next, the velocity specifying process by the velocity specifying unit 16 will be described with reference to FIGS. In the velocity specifying process, if the percussion instrument sound separation process described later performs a synchronization process between frequency spectrograms having different amplitude ratios, the frequency spectrogram converges on the smaller amplitude ratio. To get information).

  FIG. 19 is a flowchart showing the flow of velocity specifying processing by the audio signal processing apparatus 1. In the velocity specifying process, discrimination is performed for each percussion instrument (S71). First, the velocity detection range is calculated for the target percussion instrument (S72). The amplitude intensity is detected within the detection range (S73), and the amplitude intensity is converted into velocity (S74, velocity detection unit 31). Thereafter, the velocity outside the detection range is calculated based on the information obtained in S74 (S75, velocity calculating unit 32). Thereafter, S71 to S75 are repeated for each percussion instrument (S76).

  FIG. 20 is an explanatory diagram of velocity detection processing. As shown in the figure, the snare drum and hi-hat are calculated from the 3rd bar to the 6th bar for the detection range calculated in S72. This is because a cymbal or the like is often placed at the beginning of 8 bars, and an irregular rhythm pattern or sound effect such as a fill-in is often placed at the end of 8 bars. The bass drum is calculated from the second attack position to the last attack position in the eight bars. This is because in the case of a bass drum, the amplitude may be extremely large by the first attack.

  In addition, the amplitude intensity detected in S73 detects the sum of amplitude values existing in a predetermined frequency range as the amplitude intensity within the target section from the attack position of each percussion instrument. The target section is determined for each percussion instrument (several tens ms to several hundred ms). Further, the frequency range is defined as a bass drum “40 to 300 Hz”, a snare drum “600 to 3000 Hz”, and a hi-hat “6000 Hz to 16000 Hz”, similarly to the band division by the band dividing unit 13.

  In the conversion to velocity in S74, the value of 0 to 1 normalized with the largest amplitude intensity among the calculated amplitude intensity is set as the velocity within 8 bars. Note that this normalization is performed between frequency spectrograms in the same group determined by the detailed determination processing (the same percussion instrument type and the same sounding method).

  FIG. 21 is an explanatory diagram of the velocity calculation process. The calculation of the velocity in S75 is to supplement the velocity outside the detection range based on the velocity information detected in the detection range for each group. The example in the figure shows a complementary method in the case of a snare drum and a hi-hat in which the detection range is from the third bar to the sixth bar. As shown in the figure, the velocity of the detection range is calculated in units of 16th notes. Further, the velocity is averaged for each group and for each unit time (1/16 to 16/16) for the four bars (third bar to sixth bar) included in the detection range. Further, the averaged value is complemented as a supplementary value for each unit time in the bars 1, 2, 7, and 8. Although not particularly shown for the bass drum, as in the case of the snare drum or hi-hat, the value obtained by averaging the velocity information detected in the detection range for each measure is complemented as the velocity outside the detection range.

  Next, a percussion instrument sound separation process by the percussion instrument sound separation unit 18 will be described with reference to FIGS. The percussion instrument sound separation process is a process of generating frequency spectrogram information of each percussion instrument by performing a synchronization process based on the detailed discrimination information (grouping information and velocity information) obtained in the preprocessing. FIG. 22 is a flowchart showing the flow of percussion instrument sound separation processing. In the percussion instrument sound separation process, first, the composite hi-hat is separated from the continuous sound removal sound source output by the continuous sound removal unit 12 (S81). Subsequently, the bass drum is separated from the sound source (1) after the composite hi-hat separation (S82), the snare drum is separated from the sound source (2) after the bass drum separation (S83), and the sound source after the snare drum separation ( The single hi-hat is separated from 3) (S84). Further, the information that could not be detected by the attack is complemented (S85), the amplitude before the attack is removed, and spectrogram information of each percussion instrument is finally generated. Note that S81 to S84 differ only in the percussion instrument type to be processed, and the processing contents of the first specifying unit 41 to the end-of-sound determination unit 46 shown in FIG. 4 are performed in each step.

  FIG. 23A is an explanatory diagram of the composite hi-hat separation process (see S81 in FIG. 22). The composite hi-hat is based on the condition that there are hi-hats that are pronounced simultaneously with other percussion instruments such as bass drums and snare drums and hi-hats that are pronounced independently in 8 bars. In other words, even if there is a hi-hat that is sounded simultaneously with other percussion instruments such as a bass drum and a snare drum, if there is no hi-hat sounded independently, it is not regarded as a composite hi-hat. This is because bass drums and snare drums cannot be removed in the synchronous subtraction process when there is no hi-hat sounded alone. This figure is an example in which two hi-hats (reference numeral 56) indicated as “HH sung at the same time as BD” and two hi-hats indicated as “HH sung singly” are grouped as the same hi-hat. In order to isolate only the hi-hat so as not to impair the amplitude component of the bass drum struck at the same time as the hi-hat of reference numeral 56, the amplitude component of the hi-hat sung alone is used. Therefore, in the example shown in the figure, two hi-hats denoted by reference numeral 56 and two hi-hats singing alone, a total of four hi-hats, are separated as a composite hi-hat.

  On the other hand, FIG. 23B is an explanatory diagram of the bass drum separation process (see S82 in FIG. 22). As shown in the figure, in the case of a four-tone type musical composition, a bass drum and a snare drum may sound simultaneously. In this case, the bass drum has a higher number of sounds than the snare drum, and therefore has a high probability of being present alone. That is, even if an attempt is made to estimate the spectrogram shape of the snare drum without separating the bass drum, there is a high possibility that it cannot be correctly estimated because it often covers the bass drum. Another problem is that if the snare drum sound is separated while the bass drum tone remains, the bass drum sound will stand out. Therefore, it is necessary to separate the bass drum first. In the example of the figure, four bass drums denoted by reference numeral 57 are separated.

  The snare drum separation process (see S83 in FIG. 22) is not particularly shown, but the process is performed after the bass drum is separated, so that the spectrogram shape can be easily estimated. Also, the single hi-hat process (see S84 in FIG. 22) is not particularly shown. However, as with the snare drum, the process is performed after the sound already covered is separated, so that the spectrogram shape can be easily estimated. .

  Next, the synchronous subtraction process performed by the first specifying unit 41, the second specifying unit 42, and the synchronous subtracting unit 43 will be described. Percussion instruments are repeatedly played with the same waveform, but components other than percussion instruments are often played with different pitches. Therefore, by extracting the common part, it is possible to leave only the components of percussion instruments. First, a method of specifying the synchronous subtraction partner by the first specifying unit 41 and the second specifying unit 42 will be described. Synchronous subtraction is performed between those having the highest correlation value by calculating correlation values between those determined to belong to the same group in the detailed determination process. For example, as shown in FIG. 17 (b), if there are multiple frequency spectrograms of the same type in 8 bars, any frequency spectrogram (symbol 52 in the figure) is specified as the first spectrogram. To do. Also, the frequency spectrogram having the highest correlation value with the first spectrogram (symbol 53 in the figure) is specified as the second spectrogram that is the partner of the synchronous subtraction. The first spectrogram and the second spectrogram are specified for all frequency spectrograms included in one group. That is, when L frequency spectrograms are included in one group, L spectrogram identification and synchronous subtraction processing are repeated.

  FIG. 24 is an explanatory diagram of the synchronous subtraction process by the synchronous subtraction unit 43. FIG. 5A shows a state after the sizes of the spectrograms are made uniform based on velocity information for two snare drums determined as objects of synchronous subtraction. For example, the left side is the first spectrogram and the right side is the second spectrogram. As shown in FIG. 5B, when synchronous subtraction of these two snare drums is performed, a component not included in the first spectrogram as a subtraction source becomes a negative component. In addition, since components other than the snare drum component have the same size, components other than the percussion instrument can be extracted. Further, when the minus component is set to “0”, components other than the percussion instrument remain in the first spectrogram of the subtraction source. Thereafter, as shown in FIG. 4C, by subtracting components other than the percussion instrument from the first spectrogram of the subtraction source, a common component (original snare drum spectrogram shape) can be obtained.

  Note that the synchronous subtraction process is not limited to the method shown in FIG. 24 but may be a simple method in which the amplitudes of both spectrograms are compared for each frame and each bin and the smaller one is adopted.

  Next, the synchronous addition process by the synchronous adder 44 will be described. This process is performed in order to reduce the error of the spectrogram shape obtained by synchronous subtraction. FIG. 25 is an explanatory diagram of the synchronous addition process. As shown in the figure, in the synchronous addition process, the synchronous subtraction results of L frequency spectrograms existing in one group are averaged by synchronous addition. In the example of the figure, since the target of synchronous addition is four frequency spectrograms, the total value of synchronous subtraction data for the four frequency spectrograms is divided by four to obtain an average value (synchronized data). Yes. Note that the synchronous addition process is omitted when only one synchronous subtraction result exists in one group.

  Next, re-attack detection processing by the re-attack detection unit 45 will be described. In this process, the attack detection is performed based on the synchronization processed data in consideration of the case where the attack position is not accurate or the case where only the snare drum is sounded forward. FIG. 26 is a simplified flowchart showing the flow up to the re-attack detection process and an explanatory diagram thereof.

  First, a beat position analysis result is acquired from a beat position analysis application (not shown) (S91). In addition, S91 and S92 of the same figure are processes performed before acquisition of sound generation position information (refer FIG. 1). That is, the pronunciation position information in the present embodiment is generated based on the analysis result of the beat position analysis application. As shown in the explanatory diagram of S91, the beat position may be delayed in the analysis result of the beat position even if the BPM is accurate depending on the music. Therefore, in order to avoid a delay in beat positions, all beat positions are moved forward by a predetermined time (for example, 50 ms) (S92). Thereafter, synchronization processing is performed by the first identification unit 41, the second identification unit 42, the synchronization subtraction unit 43, and the synchronization addition unit 44 (S93). At this time, missed attack can be avoided by moving forward, but in the case of a music piece whose original beat position was originally detected, extra sounds (see reference numerals 61 and 62) may be added by moving forward. Of these, the reference numeral 62 is removed by the end of ringing determination (see FIG. 12) by the end of ringing determination unit 46, but the removal of the reference numeral 61 is required separately. Therefore, attack detection is performed again on the single percussion instrument after the synchronization processing (S94), and the sound before the attack is deleted (silenced) in S86 of FIG. As described above, since the re-attack detection process is performed on a single percussion instrument, it is possible to accurately detect the attack position of a song that has been advanced only by the snare drum.

  If there is no delay in the beat position, the processing of S91 and S92 can be omitted. Further, when the sound generation position information includes information indicating that “only the snare drum has been advanced”, the re-attack detection process (S94) can be omitted.

  Next, the sound end determination process by the sound end determination unit 46 will be described. This processing is performed in order to determine a rough end of the percussion instrument and delete components other than the percussion instrument, since the components other than the percussion instrument may not be deleted even if the synchronization processing is performed. Specifically, the end of ringing is determined by the two methods shown in FIG. First, in the continuous sound range detection process (see S57 in FIG. 10), when the rust flag is set, the end points according to the average value end determination method shown in FIG. This is because in the chorus, it is highly possible that the continuous sound is steady in the entire band, and therefore it is preferable to use an average value end determination method that makes it difficult for extra sound to enter. When the rust flag is not set, the end point according to the average value end determination method is used for the continuous sound range (the band in which the continuous sound exists), and FIG. The end point by the new attack judgment method shown in Fig. 1 is used. This is because the mean value end judgment method is cut off shorter than the actual sound instead of making the extra sound difficult to enter, and the new attack judgment method may cause extra sound to enter. This is because an end point similar to an actual sound can be detected (a sound can be produced until the percussion instrument disappears). In other words, it is more appropriate to use the average value end judgment method that makes it difficult for extra sounds to enter only in the continuous sound range where extra sounds may enter, and to use the new attack judgment method for other cases. The end point of the percussion instrument sound can be specified. Although the bass drum ringing end determination is illustrated in FIG. 12, the sounding end determination process is similarly performed for the snare drum and the hi-hat.

  By the way, in the sounding end determination process, since the sounding end is determined for each frequency bin, the sounding end point may be extremely shorter or longer than the surrounding frequency bins. When such an extreme end point is detected, the percussion instrument sound to be separated is deteriorated. For this reason, it is necessary to align the ringing end points.

  FIG. 27 is an explanatory diagram of a ringing end correction process. As shown in FIG. 6A, this process is performed when the end of the ringing is extremely short and the end of the long ringing is extremely short as shown in FIG. It is a process that adjusts the end of a long ringing around. FIG. 5C shows a method for matching the end of the ringing around. As shown in the figure, the entire frequency band is divided into 1/3 octave widths, and the end points of sounding are aligned to the median in 1/3 octave width units. Note that the setting of the number of divided bands and the range of each band can be changed.

  Next, a method for supplementing information that could not be attack detected (S85 in FIG. 22) will be described. In the case of a percussion instrument with a very weak attack, there may be a component that does not exceed the “threshold value regarded as an attack” in the re-attack detection process (see S94 in FIG. 26). This is because the “threshold value regarded as an attack” differs between the process of detecting the pronunciation position information in FIG. 1 and the percussion instrument sound separation process. Therefore, the mode value of the attack value indicating how much the beat position obtained from the beat position analysis application is deviated from the true attack position is calculated, and the information that could not be detected is complemented. The location where the attack could not be detected is less than the “threshold value to be regarded as an attack” and is a very weak attack. However, since the pronunciation position of the percussion instrument is known from the pronunciation position information, this portion is complemented. For example, a case where five types of hi-hats are sounding is assumed. Assume that the attack value (attack deviation time) is calculated as 5 ms for hi-hat 1, 10 ms for hi-hat 2, 10 ms for hi-hat 3, and 8 ms for hi-hat 4, and hi-hat 5 does not satisfy the attack threshold. At this time, 5 ms and 8 ms appear only once, but 10 ms appears twice. Therefore, even when the attack value of hi-hat 5 is unknown, it is assumed that 10 ms appears most frequently.

  Next, application examples of the separated percussion instrument sounds will be described. FIG. 28 is an explanatory diagram regarding adjustment of percussion instrument sounds. As shown in the figure, each volume other than the bass drum, snare drum, hi-hat, and three sounds may be adjusted using an operator such as a rotary type operator 71. Further, not the volume but the separation rate (generation rate) of each percussion instrument sound may be adjustable. In this case, the minimum value of the adjustable separation rate may be set to 0 (zero).

  In addition, different effects may be applied to each sound other than the bass drum, snare drum, hi-hat, and three sounds. Further, the effect applying rate (processing amount) may be adjustable by the user. Various effects such as delay, reverb, echo, and other effects used in DJ equipment effectors can be applied as effects. As an operation method, for example, when the rotary type operation element 71 corresponding to the bass drum is rotated to the right, the number of bass drum sounds is gradually increased (added with a delay), and when rotated to the left, the bass For example, the number of drum sounds can be gradually attenuated. The type of the operation element is not limited to the rotary type operation element 71, and any type such as a fader type operation element or a touch panel may be used.

  Further, as shown in FIG. 29, the separated percussion instrument sound may be displayed as a musical score. That is, the discrimination result of the bass drum, snare drum, and hi-hat may be converted to MIDI (Musical Instrument Digital Interface) and used as the drum score 72. In this case, the hi-hat may be divided into an open hi-hat and a closed hi-hat, and may be displayed by percussion instrument type (by group). Further, instead of the snare drum, a hand clap may be displayed as a musical score. The hand clap can estimate and separate the spectrogram shape by the same processing steps as the snare drum.

  Although not particularly shown, the drum may be sounded with MIDI to switch the timbre. That is, another sound (such as an acoustic drum) may be output at the timing of each percussion instrument sound attack. Alternatively, the percussion instrument sounds may be sampled and output according to the sequence input by the user (or at the output timing specified by the user).

  As described above, according to the present embodiment, the percussion instrument sound separation unit 18 estimates the frequency spectrogram by the synchronization process, so that it is not necessary to use a template. Therefore, the spectrogram shape for each percussion instrument can be accurately estimated regardless of the music. Further, since the continuous sound removing unit 12 removes the continuous sound component that has continued for a predetermined time or longer, the spectrogram shape by the percussion instrument sound separating unit 18 can be estimated more accurately.

  Further, since the detailed discrimination sound processing unit 14 removes unnecessary sounds for each percussion instrument as pre-processing of the detailed discrimination process, the correct discrimination rate of the detailed discrimination unit 15 can be increased. Further, since the detailed discrimination sound processing unit 14 specifies the search section from the percussion instrument attack position and subtracts the amplitude value at the predetermined position of the search section, unnecessary sounds can be removed with simple processing. . Moreover, since the method of extracting the amplitude value differs for each percussion instrument, the target percussion instrument sound can be extracted more accurately. Further, in the processing of the bass drum, the end of the ringing is determined, and when the ringing is completed, the rest of the ringing is set to zero, so that unnecessary sounds can be reliably removed. Further, since the detailed discrimination section 15 groups frequency spectrograms on the condition that the same percussion instrument type and the same sounding style are used, the accuracy of the synchronization processing in the percussion instrument sound separation section 18 at the subsequent stage can be improved.

  Further, since the velocity specifying unit 16 specifies the velocity of each percussion instrument, the groove determining unit 17 in the subsequent stage can accurately and easily determine the groove of the music. In addition, in order to specify the velocity, some sections of the 8 measures of the music are less susceptible to noise (bass drum is the second attack to the last attack, snare drum and hi-hat are the third measure to 6 Since velocity is detected for the bar), an accurate detection result can be obtained. Further, the detection results are averaged for each group and for each unit time, and the averaged value is complemented to a section other than a part of the eight bars. Can be accurately identified. Further, by accurately specifying the velocity, the accuracy of the synchronization processing in the percussion instrument sound separation unit 18 can be increased.

  Note that the following modifications and application examples can be employed. For example, in the above embodiment, the sound generation position information is acquired from the outside. However, the sound signal processing apparatus 1 may analyze music and generate sound generation position information. Further, the user may manually input the pronunciation position information.

  Further, as shown in FIG. 30A, the bass drum processing unit 21 of the above embodiment performs a process of subtracting the amplitude data 51a in the last frame of the search section (t1 to t2). As shown in (b), the amplitude data 51b in the first frame of the search section (t1 to t2) may be subtracted. The former case (in the case of the above embodiment) is effective when the pitch is switched simultaneously with the bass drum, as shown in FIG. In the latter case, as shown in FIG. 5B, it is effective when the bass sound continues to sound before the bass drum sounds. In this case, if the last amplitude data in the search section is subtracted, the pitch may change or the bass sound itself may be attenuated. In addition, the position of the frame to be subtracted (the position extracted by the extraction unit 21d) may be set by the user based on the start position, end position, or attack position of the search section, and the frame may be set according to the music analysis result. The position may be variable.

  As a modification of the bass drum processing unit 21, the extraction unit 21d generates a predetermined number (however, 2 or more) frames from the beginning of the search section or a predetermined number (however, 2 or more) frames from the end of the search section. The average amplitude value of a predetermined number of extracted frames may be subtracted from all the frames included in the search section by the first processing unit 21e. In this case, the predetermined number is preferably the same number as the overlap number (“4” in the case of four overlaps).

  Further, as a modification of the first processing unit 21e in the bass drum processing unit 21, processing other than subtraction may be performed. That is, any calculation method may be used as long as the voice data included in the search section is processed based on the amplitude value extracted by the extraction unit 21d. Even when the subtraction processing is performed, the subtraction ratio may be subtracted at a predetermined ratio such as 80% or 50% instead of 100%.

  Further, as an application example of the detailed discrimination sound processing unit 14, processing may be performed on sounds other than percussion instrument sounds. For example, as shown in FIG. 31, when vocals are played in accordance with piano accompaniment, amplitude data at positions where no vocal fundamental tone exists in each frequency band (for example, amplitude data indicated by reference numeral 54 or reference numeral 55). By extracting and subtracting, it is possible to extract only vocals.

  Further, the velocity specifying unit 16 of the above embodiment specifies (detects and calculates) the velocity in units of sixteenth notes obtained by dividing one measure into sixteen parts, but the specific unit (number of divisions) is arbitrary. Further, the specific unit may be varied according to the music (according to information such as sound generation position information, music genre, music BPM (Beats Per Minute), rhythm, etc.).

  Further, the percussion instrument sound separation unit 18 of the above-described embodiment performs spectrogram shape estimation and percussion instrument sound separation for three percussion instruments (bass drum, snare drum, hi-hat). The present embodiment can be applied. The present embodiment can also be applied to rhythm instruments other than percussion instruments or instruments other than rhythm instruments.

  In the above embodiment, the predetermined sound generation period is eight bars, but a longer / shorter period may be used. Further, one music piece may be set as a predetermined pronunciation period.

  Moreover, it is possible to provide each part and each function in the audio | voice signal processing apparatus 1 shown in said each embodiment as a program (application). Further, the program can be provided by being stored in various recording media (CD-ROM, flash memory, etc.). That is, a program for causing a computer to function as each unit of the audio signal processing apparatus 1 and a recording medium on which the program is recorded are also included in the scope of the right of the present invention. In addition, the audio signal processing device 1 can be appropriately changed without departing from the gist of the present invention, such as being realized by a server (cloud computing) on a network.

  DESCRIPTION OF SYMBOLS 1: Audio | voice signal processing apparatus 11: FFT part 12: Continuous sound removal part 13: Band division part 14: Sound processing part for detailed determination 15: Detailed determination part 16: Velocity specific part 17: Groove determination part 18: Percussion instrument sound separation part 21: Bass drum processing unit 21a: Sound generation position information acquisition unit 21b: Search section specifying unit 21c: End of sound determination unit 21d: Extraction unit 21e: First processing unit 21f: Second processing unit 22: Snare drum processing unit 23: Hi-hat Processing unit 31: Velocity detection unit 32: Velocity calculation unit 41: First identification unit 42: Second identification unit 43: Synchronization subtraction unit 44: Synchronization addition unit 45: Re-attack detection unit 46: Sound end determination unit 47: Sound source generation Part 48: Sound source separation part

Claims (12)

A first specifying unit for specifying a first spectrogram which is a frequency spectrogram of an arbitrary instrument from a predetermined sound generation section;
A second specifying unit that specifies a second spectrogram that is a frequency spectrogram of the same musical instrument as the arbitrary musical instrument based on a correlation value with the first spectrogram from the predetermined sounding section;
An audio signal processing apparatus, comprising: a synchronous subtractor that extracts a common component of the first spectrogram and the second spectrogram.   The second specifying unit specifies, as the second spectrogram, a frequency spectrogram having the highest correlation value with the first spectrogram when a plurality of frequency spectrograms of the same instrument as the arbitrary instrument exist in the predetermined tone generation section. The audio signal processing apparatus according to claim 1.   When there are L frequency spectrograms of the arbitrary musical instrument in the predetermined sound generation interval (where L is an integer satisfying L ≧ 2), the L common components extracted by the synchronous subtraction unit are averaged. The audio signal processing apparatus according to claim 2, further comprising a synchronous adder that calculates a common spectrogram.   A sound source for generating a synchronization-processed sound source of the arbitrary musical instrument by replacing the L frequency spectrograms of the arbitrary musical instrument existing in the predetermined sound generation section with the common spectrogram calculated by the synchronous addition unit The audio signal processing apparatus according to claim 3, further comprising a generation unit. A sound source separation unit that separates a synchronization-processed sound source of the arbitrary musical instrument generated by the sound source generation unit from an arbitrary music piece;
If the arbitrary music contains a plurality of instrument sounds,
For each instrument sound, performing instrument sound separation processing including processing of the first specifying unit, the second specifying unit, the synchronization subtracting unit, the synchronization adding unit, the sound source generating unit, and the sound source separating unit. The audio signal processing device according to claim 4, wherein When the plurality of instrument sounds are bass drums, snare drums, hi-hats, and there is a hi-hat that is sounding alone and a hi-hat that is sounding simultaneously with another percussion instrument in the arbitrary music piece,
6. The instrument sound separation process of claim 5, wherein the synchronized sound source is separated in the order of a hi-hat sounded simultaneously with another percussion instrument, a bass drum, a snare drum, and a hi-hat sounded independently. The audio signal processing device described. For a frequency band defined for each percussion instrument, further comprising a velocity specifying unit for specifying a velocity for each percussion instrument for each unit time obtained by equally dividing the predetermined sound generation section,
The voice according to claim 6, wherein the synchronous subtraction unit extracts the common component after aligning amplitude values of the first spectrogram and the second spectrogram based on a velocity for each percussion instrument. Signal processing device. A detailed discriminating section for grouping a plurality of frequency spectrograms existing in the predetermined sound generation section on condition that the same percussion instrument type and the same sounding method are used,
The audio signal processing apparatus according to claim 6 or 7, wherein the second specifying unit specifies the second spectrogram from one or more frequency spectrograms belonging to the same group as the first spectrogram. A pronunciation position information acquisition unit for acquiring pronunciation position information indicating the pronunciation positions of the three percussion instruments;
When it is known from the sound generation position information that there are a plurality of frequency spectrograms of the same percussion instrument as the first spectrogram in the predetermined sound generation section, the detailed determination unit determines the first spectrogram of the plurality of frequency spectrograms. 9. The audio signal processing apparatus according to claim 8, wherein an average value of correlation values with respect to is calculated, and a frequency spectrogram of correlation values exceeding the average value is classified as the same group as the first spectrogram. A continuous sound removing unit that extracts a continuous sound component that continues for a predetermined time or more based on amplitude spectrum information obtained by performing a frequency Fourier transform on the sound signal of the arbitrary music piece, and removes the continuous sound component from the sound signal; Prepared,
The said 1st specific part and the said 2nd specific part identify the said 1st spectrogram and the said 2nd spectrogram after the said continuous sound component is removed, The any one of Claim 5 thru | or 9 characterized by the above-mentioned. The audio signal processing apparatus according to 1. A first specifying step of specifying a first spectrogram which is a frequency spectrogram of an arbitrary instrument from a predetermined sound generation section;
A second specifying step of specifying a second spectrogram which is a frequency spectrogram of the same musical instrument as the arbitrary instrument based on a correlation value with the first spectrogram from the predetermined sounding section;
A method for controlling an audio signal processing device, comprising: performing a synchronous subtraction step of extracting a common component of the first spectrogram and the second spectrogram.   The program for making a computer perform each step in the control method of the audio | voice signal processing apparatus of Claim 11.

Images