JP6257537B2 - Saliency estimation method, saliency estimation device, and program - Google Patents

Description

  The present invention relates to a saliency estimation method, a saliency estimation apparatus, and a program for estimating the degree of conspicuousness of sound in a part of a time interval of an input signal.

  Conventionally, a model (auditory saliency map) for calculating the degree of conspicuousness in the time-frequency domain based on the spectral structure of sound has been proposed (see Non-Patent Document 1).

C. Kayser, C. I. Petkov, M. Lippert, N. K. Logothetis, "Mechanisms for Allocating Auditory Attention: An Auditory Saliency Map", Current Biology, 2005, Volume 15, Issue 21, Pages 1943-1947.

  According to the Auditory Saliency Map, it is possible to estimate the conspicuousness of a sound at a specific time frequency by calculating the intensity and its time-frequency contrast with respect to the sound spectrogram.

  Since this model performs calculations based on the spectral structure of the sound, the same degree of saliency is evaluated for the same sound regardless of the context in which a specific sound is presented. Therefore, even if the sound is the same, it is not possible to sufficiently express the saliency based on the time-series pattern or context, such as how the saliency increases due to the presentation at an unexpected timing.

  Therefore, an object of the present invention is to provide a saliency estimation method capable of estimating the degree of conspicuousness of a target sound based on the unpredictability of a time-series pattern.

  The present invention is a saliency estimation method for estimating the degree of conspicuousness of sound in a part of a time interval of an input signal, and includes a similar partial interval detection step, a predicted distribution generation step, and a saliency estimation step. Among the time intervals of the input signal, the prediction interval is a time interval that is a target of estimation of the saliency that is the degree of conspicuousness of the sound, and the reference interval is a time interval that has a predetermined time width immediately before the prediction interval. The accumulation interval is a time interval before the reference interval, the estimation target signal is the feature amount of the input signal corresponding to the prediction interval, and the reference signal is the feature amount of the input signal corresponding to the reference interval.

  The similar partial section detection step detects one or more similar partial sections that are similar to the reference signal from the accumulation section. The prediction distribution generation step generates one or more prediction distributions that are distributions based on the feature amount of the input signal corresponding to a predetermined time interval immediately after the detected similar partial interval. The saliency estimation step estimates the saliency of the input signal corresponding to the prediction interval based on the generated prediction distribution and the estimation target signal.

  According to the saliency estimation method of the present invention, it is possible to estimate the degree of conspicuousness of the target sound based on the unpredictability of a time-series pattern.

1 is a block diagram illustrating a configuration of a saliency estimating apparatus according to Embodiment 1. FIG. 3 is a flowchart showing the operation of the saliency estimating apparatus according to the first embodiment. The figure explaining the definition of an accumulation area, a reference area, a prediction area, an accumulation | storage partial area, and a similar partial area. The figure which shows the example which calculated | required the similar partial area by setting an accumulation area, a reference area, and a prediction area with respect to actual music. The figure which shows the example which calculated | required saliency with respect to each of two types of waveforms based on actual music.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the structure part which has the same function, and duplication description is abbreviate | omitted.

  Hereinafter, a saliency estimation apparatus according to Embodiment 1 that estimates the degree of conspicuousness of sound based on the unpredictability of a time-series pattern of an acoustic signal will be described with reference to FIGS. 1 and 2. Note that the degree of conspicuousness of sound is also referred to as saliency. FIG. 1 is a block diagram showing the configuration of the saliency estimating apparatus 1 of this embodiment. FIG. 2 is a flowchart showing the operation of the saliency estimating apparatus 1 of the present embodiment. The saliency estimating apparatus 1 of the present embodiment is an apparatus that evaluates saliency in a specific time section of an input signal that is a specific acoustic signal, and more specifically, a general having a CPU (Central Processing Unit) and a memory. Realized on a simple computer. As shown in FIG. 1, the saliency estimating apparatus 1 of this embodiment includes a feature amount calculation unit 11, a section similarity calculation unit 12, a similar partial section detection unit 13, a predicted distribution generation unit 14, and a saliency level. An estimation unit 15 is included. Hereinafter, the operation of each component will be described in detail.

<Feature Quantity Calculation Unit 11 (Step S11)>
The feature quantity calculator 11 calculates and outputs a feature vector for each frame by performing a time-frequency analysis over the entire input signal (S11).

  For example, the feature amount calculation unit 11 converts the entire input signal into a power spectrogram based on an auditory filter. The auditory filter is a band filter group whose center frequency continuously changes. The feature amount calculation unit 11 performs frequency analysis on the signal sound, with each filter serving as a bandpass filter (here, a gamma tone filter) having a different bandwidth.

When the total number of auditory filters is C, the center frequency f _k (k = 1, 2,..., C, f _k represents the center frequency of the _kth filter) of each filter is the maximum frequency of the target frequency band. (Usually the Nyquist frequency) is f _max and the minimum frequency is f _min .

It is expressed. However, Q _e represents an asymptotic value of the Q value (= center frequency / bandwidth) in the high frequency band, and w ₀ represents a lower limit value of the bandwidth in the low frequency band, both of which are constants. At this time, the bandwidth ERB _k of each filter is

It is expressed. The obtained filter output has the same sampling frequency as that of the signal sound, but for the purpose of dimension reduction and noise reduction, by taking a moving average for each frame width ΔT (about several ms) and section width ΔT / 2, it is a frame unit. Convert to. In the following, the i-th (i = 1, 2,..., C) auditory filter output at frame number t is expressed as p _i [t], and the output (C-dimensional vector) of the entire auditory filter at that time point is

It expresses. Hereinafter, p [t] is referred to as a feature vector at frame number t. The feature quantity calculator 11 may calculate the feature vector by a technique such as NMF (non-negative matrix factorization). Further, the feature quantity calculation unit 11 may calculate a feature vector by a simple spectrogram.

<Section Similarity Calculation Unit 12 (Step S12)>
Hereinafter, each section of the input signal will be defined with reference to FIG. FIG. 3 is a diagram for explaining the definition of the accumulation interval, the reference interval, the prediction interval, the accumulation partial interval, and the similar partial interval. As shown in FIG. 3, a time interval in which a prediction interval is a saliency estimation target, a reference interval immediately before the prediction interval and a predetermined time width, and an accumulation interval before the reference interval (Overall time interval), the accumulation partial interval is defined as a time interval that is a partial interval of the accumulation interval and has the same time width as the reference interval. In addition, the feature quantity of the input signal corresponding to the prediction section (entire feature quantity) as the estimation target signal, the feature quantity of the input signal (entire feature quantity) corresponding to the reference section as the reference signal, and the input corresponding to the accumulation section as the accumulation signal The feature amount of the signal (entire feature amount) and the accumulated partial signal are defined as the feature amount (entire feature amount) of the input signal corresponding to the accumulated partial interval.

  The section similarity calculation unit 12 calculates a section similarity, which is the similarity between the stored partial signal of the stored signal and the reference signal, based on the feature vector calculated in step S11 (S12).

  In the following, as shown in FIG. 3, the frame number of the accumulation section is 1 to T, the frame number of the reference section is T + 1 to T + N, and the frame number of the prediction section is T + N + 1 to T + N + M. However, since the accumulation interval needs to be longer than the reference interval, T> N.

The interval similarity calculation unit 12 calculates the angle of the power spectral density for all the combinations of frames of the feature vector, and defines it as the similarity between frames. The similarity S (t ₁ , t ₂ ) between the feature vectors of frame numbers t ₁ and t ₂ is

Is calculated.

  In order to extract a partial section similar to the reference section in the accumulation section, the section similarity calculation unit 12 has the same length (number of frames N) as the reference section based on the similarity in frame units calculated above. Calculate the similarity in units. The section similarity SIM (t) between the reference section and the storage partial section (frame number t to t + N−1) is:

Is calculated. SIM (t) is calculated over 1 ≦ t ≦ T. The calculation of the interval similarity may be performed by a method described in Japanese Patent No. 4327202, for example.

<Similar Partial Section Detection Unit 13 (Step S13)>
The similar partial section detection unit 13 detects one or more similar partial sections, which are partial sections similar to the reference signal, from the storage section based on the section similarity output by the section similarity calculation unit 12 (S13). The similar partial section detection unit 13 extracts D frame numbers having a high section similarity SIM (t) value in order from the top in 1 ≦ t ≦ T. D is a constant and it is desirable to extract several to several tens.

  Actually, there is a possibility that partial sections with high similarity overlap and concentrate on a specific time range, so set a certain threshold, exclude adjacent frame numbers within the threshold range, and One section can also be extracted. The D partial sections thus detected are referred to as similar partial sections. The similar partial section detection may be performed by a method described in, for example, Japanese Patent No. 4327202.

  FIG. 4 shows an example in which similar partial sections are obtained by setting a storage section, a reference section, and a prediction section for actual music. In the example of FIG. 4, as a partial section similar to the reference signal of the reference section set in the vicinity of 5 to 6 seconds, about 1 to 2 seconds (similar partial section 1) in the accumulation section, 3 to 4 seconds (similar) Partial section 2) has been detected.

<Prediction distribution generation unit 14 (step S14)>
The prediction distribution generation unit 14 generates a prediction distribution that is a distribution for predicting the feature vector of the prediction section based on the feature amount of the input signal corresponding to the predetermined time section immediately after the detected similar partial section (S14). ).

  Specifically, the prediction distribution generation unit 14 generates a prediction distribution for each element of the C-dimensional feature vector for each frame in the prediction section. The predicted distribution is output as D distributions calculated based on each of the D similar partial sections detected by the similar partial section detection unit 13.

In the frame number T + N + t of the prediction interval, the prediction distribution generated from the similar partial interval having the start frame number L _d is a multidimensional normal distribution N (μ that represents the mean μ _d and the variance-covariance matrix Σ _d by the following equation: _d, and Σ _d).

Where v _d is the variance defined for each auditory filter,

And σ ² is a C-dimensional vector in which the variance values of each element in the accumulation interval are arranged, and takes a constant value regardless of the similarity. The second term represents a coefficient that depends on the similarity. When the similarity degree approaches 1, the variance value approaches 0, and the prediction concentrates around the average value with a high probability. When the similarity is 0, the variance value (when t = 0) matches the variance value σ ² in the accumulation interval. This is because the expected value of the difference (absolute value) between the predicted distribution and the observed value is the expected value of the difference (absolute value) between any two points in the past time series, i.e. Rely on being equal to the standard deviation. The third term represents time decay. The variance value increases exponentially with the passage of time, which means that the predicted distribution approaches a uniform distribution. γ is a constant representing the degree of attenuation and can be set to any positive value.

  As can be seen from the above equation, the average value of the prediction distribution matches the behavior of the input signal in the M frame immediately after the similar partial section. The variance value of the prediction distribution changes depending on the similarity SIM (d) with the reference section of each similar partial section. In addition to the above definition of variance, add the terms that increase the variance value of the predicted distribution according to the time difference between the similar partial interval and the reference interval, and consider the “forgetting” factor related to prediction. You can also.

<Saliency estimation unit 15 (step S15)>
The saliency estimation unit 15 determines the degree of conspicuity (saliency) of the acoustic signal (input signal) corresponding to the prediction section based on the comparison between the prediction distribution generated by the prediction distribution generation unit 14 and the actual estimation target signal. Estimate (S15).

  Specifically, the saliency estimation unit 15 calculates elements of saliency based on the respective distributions based on the probability that the estimation target signal appears in the D prediction distributions output from the prediction distribution generation unit 14. . For the saliency element, the saliency is output by taking a weighted average corresponding to the similarity value between each similar partial section and the reference section.

Elements of saliency for (like parts based on interval of the start frame number L _d) estimated for the target signal p [T + N + t] , d -th prediction distribution, respectively

Is calculated as With respect to the above values, the saliency z [T + N + t] is defined by taking a weighted average corresponding to the similarity between the d-th similar partial section and the reference section.

Where A is a normalization factor,

It is.

  Up to this point, the start frame number of the prediction interval has been fixed to T + N + 1. For example, the saliency is calculated while changing the start frame number of the prediction interval over the entire range of the input signal, and the accumulated value of the saliency is obtained. Therefore, the degree of saliency may be defined.

  FIG. 5 shows an example in which the saliency is obtained for actual music. 5A and 5B show examples in which the same musical tone signal is listened for the first time and the second time, respectively, with the vertical axis representing signal intensity and the horizontal axis representing time. C and D in FIG. 5 exemplify the transition of the saliency of the two types of music signals, with the ordinate representing the saliency with respect to each of the tone signals shown in FIGS. 5A and 5B and the abscissa representing the time. The time before the time indicated by the vertical line (before about 5.8 s) represents the prediction target section, and the time period before the time indicated by the vertical line (before about 5.8 s) represents the reference section (and the accumulation section). The solid line waveforms in FIGS. 5A and 5B are actual musical tone signals, and the dotted line represents the predicted musical tone signal (average value of the predicted distribution). In the first listening, the musical tone signal in the prediction target section is not correctly predicted, and a pattern as shown in the upper right of FIG. 5A (a pattern that repeatedly appeared in the accumulation section) is predicted. As shown in C, the saliency is calculated as a relatively large value. In the second listening, since the stored signal contains almost the same signal as the musical tone signal in the prediction target section, the pattern as shown in the upper right of FIG. 5B is correctly predicted. As shown in FIG. 5, the saliency is calculated as a relatively small value.

<Effect>
According to the saliency estimating apparatus 1 of the present embodiment, the saliency in a specific time section of an input signal that is a specific acoustic signal can be evaluated by the above configuration.

  According to Non-Patent Document 1, saliency of auditory stimulation can be evaluated in time series based on the spectral characteristics of sound. However, in this model, calculations based on the spectral structure of the sound in a short time window are performed, so the same saliency is evaluated for the same sound regardless of the context. Therefore, even for the same sound, a change in saliency based on a temporal pattern change, such as an increase in saliency due to presentation at an unexpected timing, cannot be expressed sufficiently. Since the calculation based on the unpredictability of time-series patterns is performed in this method, the degree of conspicuousness of sounds based on pattern changes is estimated for acoustic signals composed of repetitions and deviations such as music. can do.

  When a future value from a specific time point is predicted based on past information for a time series signal, it is conceivable to use a time series prediction method such as an AR model. However, in the time series prediction method for steady processes as in the AR model, it is difficult to reproduce a specific past pattern because all past contributes equally to generation of the model. In this method, it is assumed that the time series development from a given point of time behaves in the same way as the past signal pattern, and the certainty of behavior (variance of probability distribution) is the past. The model correlates with the similarity. As a result, a statistical pattern can be predicted for a time-series signal that is a complex non-stationary process such as music.

<Supplementary note>
The apparatus of the present invention includes, for example, a single hardware entity as an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, and a communication device (for example, a communication cable) capable of communicating outside the hardware entity. Can be connected to a communication unit, a CPU (Central Processing Unit, may include a cache memory or a register), a RAM or ROM that is a memory, an external storage device that is a hard disk, and an input unit, an output unit, or a communication unit thereof , A CPU, a RAM, a ROM, and a bus connected so that data can be exchanged between the external storage devices. If necessary, the hardware entity may be provided with a device (drive) that can read and write a recording medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.

  The external storage device of the hardware entity stores a program necessary for realizing the above functions and data necessary for processing the program (not limited to the external storage device, for example, reading a program) It may be stored in a ROM that is a dedicated storage device). Data obtained by the processing of these programs is appropriately stored in a RAM or an external storage device.

  In the hardware entity, each program stored in an external storage device (or ROM or the like) and data necessary for processing each program are read into a memory as necessary, and are interpreted and executed by a CPU as appropriate. . As a result, the CPU realizes a predetermined function (respective component requirements expressed as the above-described unit, unit, etc.).

  The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. In addition, the processing described in the above embodiment may be executed not only in time series according to the order of description but also in parallel or individually as required by the processing capability of the apparatus that executes the processing. .

  As described above, when the processing functions in the hardware entity (the apparatus of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entity should have are described by a program. Then, by executing this program on a computer, the processing functions in the hardware entity are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto-Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims (5)

A saliency estimation method for estimating the degree of conspicuousness of sound in a part of a time interval of an input signal,
Of the time interval of the input signal,
Let the prediction interval be the time interval for which the saliency, which is the conspicuousness of the sound, is to be estimated,
The reference interval is a time interval immediately before the prediction interval and having a predetermined time width,
The accumulation interval is a time interval before the reference interval,
The estimation target signal is a feature amount of the input signal corresponding to the prediction interval,
Let the reference signal be the feature quantity of the input signal corresponding to the reference section,
A similar partial section detecting step of detecting one or more similar partial sections that are similar to the reference signal from the accumulation section;
A prediction distribution generation step of generating one or more prediction distributions that are distributions based on the feature amount of the input signal corresponding to a predetermined time interval immediately after the detected similar partial interval;
A saliency estimation step of estimating the saliency of the input signal corresponding to the prediction interval based on the generated prediction distribution and the estimation target signal;
A saliency estimation method including: The saliency estimation method according to claim 1,
The saliency estimation step includes:
A saliency estimation method for estimating the saliency based on an appearance probability that the estimation target signal appears in one or more prediction distributions generated in the prediction distribution generation step. The saliency estimation method according to claim 2,
A saliency estimation method for changing a saliency based on an appearance probability that the estimation target signal appears according to a similarity between the similar partial section and the reference section. A saliency estimation device for estimating the degree of conspicuousness of sound in a part of a time interval of an input signal,
Of the time interval of the input signal,
Let the prediction interval be the time interval for which the saliency, which is the conspicuousness of the sound, is to be estimated,
The reference interval is a time interval immediately before the prediction interval and having a predetermined time width,
The accumulation interval is a time interval before the reference interval,
The estimation target signal is a feature amount of the input signal corresponding to the prediction interval,
Let the reference signal be the feature quantity of the input signal corresponding to the reference section,
A similar partial section detector that detects one or more similar partial sections that are similar to the reference signal from the accumulation section;
A prediction distribution generation unit that generates one or more prediction distributions that are distributions based on the feature amount of an input signal corresponding to a predetermined time interval immediately after the detected similar partial interval;
A saliency estimation unit that estimates the saliency of the input signal corresponding to the prediction interval based on the generated prediction distribution and the estimation target signal;
A saliency estimation device including:   A program for causing a computer to execute each step of the saliency estimation method according to any one of claims 1 to 3.

Images