KR20230044574A - Data augmentation method using fundamental freuqency obtained by dj transform - Google Patents

Abstract

As being related to in generating training data of machine learning for speech recognition or speaker recognition by enabling each step to be performed by a computer processor, a data augmentation method according to an embodiment of the present invention comprises: a step of extracting at least one fundamental frequency of speech data; and a step of generating harmonic waves of the fundamental frequency as training data. Therefore, the present invention is capable of improving a performance of machine learning.

Description

Data augmentation method using fundamental frequency obtained through DJ conversion {DATA AUGMENTATION METHOD USING FUNDAMENTAL FREUQENCY OBTAINED BY DJ TRANSFORM}

The present invention relates to a data augmentation method for generating training data for machine learning in the field of speech recognition or speaker recognition.

Speech recognition is a technology in which a computer interprets the voice language spoken by a person and converts the contents into text data, and speaker recognition refers to a technology that distinguishes a speaker by analyzing a voice print, which can be called a 'voice fingerprint'. It is divided into speaker identification and speaker verification. Speaker identification is a technology to find a speaker with a voice most similar to the input utterance among registered voiceprint data, and speaker verification is a technology to determine whether the input voiceprint and the registered voiceprint are the same person's voiceprint. Recently, voice and speaker recognition technologies have been improved through deep learning models, which are a type of machine learning, and interest in technologies for more successful machine learning is deepening.

In general, in machine learning, data augmentation techniques are applied to increase the amount of training datasets and improve their diversity. In machine learning, data augmentation is a method of increasing the amount of a training dataset, and a method of adding a modified copy of an original training dataset or adding newly synthesized data from an original training dataset is used.

A widely used data augmentation technique in the voice domain is a data augmentation method that uses the MUSAN dataset announced in 2018 to add noise to audio and apply reverberation with RIR (Room Impulse Response). However, since this data augmentation method only augments data by four simple patterns of echo, buzz, music, and noise, performance improvement is limited when noise from a new environment is added.

Registered Patent No. 10-2158743

Embodiments of the present invention are intended to propose new data augmentation methods for machine learning systems that are robust to new environments.

The data augmentation method according to an embodiment of the present invention is a data augmentation method in which each step is performed by a computer processor and generates machine learning training data for speech recognition or speaker recognition,

(a) extracting at least one fundamental frequency of voice data; and

(b) generating harmonics of the fundamental frequency as training data.

In step (a),

(a-1) With respect to the input of voice data, the vibration motion of a plurality of springs having different natural frequencies is modeled, respectively, to calculate the estimated pure tone amplitude and phase according to the natural frequencies, and the natural frequency of each of the plurality of springs is calculated. generating a DJ-converted spectrogram representing the estimated pure tone amplitude according to a corresponding frequency and a plurality of time points; and

(a-2) extracting the fundamental frequency based on a moving average of the estimated pure tone amplitude or a moving standard deviation of the estimated pure tone amplitude for each natural frequency of the DJ transform spectrogram. .

In the step (a-1),

estimating a steady-state expected amplitude, which is a convergence value of amplitudes of each of the plurality of springs in a stable state, based on amplitudes of two time points of a unique one-cycle interval of each of the plurality of springs; and

The method may include calculating the estimated pure tone amplitude based on the predicted pure tone amplitude, which is the amplitude of the input sound estimated based on the expected stable state amplitude.

The estimated pure tone amplitude may be the steady state expected amplitude or the predicted pure tone amplitude.

In the step (a-2),

(a-21) calculating a degree of fitness for the fundamental frequency based on a moving average of the estimated pure-tone amplitudes or a moving standard deviation of the estimated pure-tone amplitudes in the DJ transform spectrogram; and

(a-22) calculating the maximum value of the fitness for the fundamental frequency at each of the plurality of points in time, and extracting the fundamental frequency based on the calculated maximum value of the fitness for the fundamental frequency.

The degree of fitness for the fundamental frequency may be proportional to a moving average of the estimated pure-tone amplitude or inversely proportional to a moving standard deviation of the estimated pure-tone amplitude.

In the step (a-22),

At each of the plurality of points in time, the top N (N is an integer of 2 or more) of the fundamental frequency conformity are extracted, the values corresponding to the natural frequencies corresponding to the N are set to "1", and the remaining values are set to "0". Black and white spectrogram generation step set to ";

an average black-and-white spectrogram generating step of calculating an average of the black-and-white spectrogram for an area of the same size including each point of the black-and-white spectrogram; and

The step of extracting a maximum value of the average black-and-white spectrogram at each of the plurality of time points may be included.

In the step (a-22), at each of the plurality of points in time, the difference between the natural frequencies corresponding to adjacent maxima of the average black-and-white spectrogram and the natural frequencies corresponding to the maxima of the average black-and-white spectrogram are determined at the lowest frequency. Based on the method, a step of extracting a candidate fundamental frequency may be further included.

The step (a-22) calculates the time average of the black-and-white spectrogram-based fundamental frequencies set for a predetermined time interval including adjacent time points, and is approximately equal to the value obtained by multiplying the time average by positive integers less than or equal to the predetermined value. A value obtained by dividing a frequency having the largest average black-and-white spectrogram among the frequencies belonging to the first frequency set by a positive integer multiplied when setting the first frequency set , and may further include setting a black-and-white spectrogram-based basic frequency.

In the step (a-22), among the candidate fundamental frequencies for the plurality of viewpoints, the candidate fundamental frequency of a viewpoint having the smallest moving variance of the difference between the candidate fundamental frequencies of adjacent viewpoints is selected, A step of setting the basic frequency based on the black and white spectrogram of the viewpoint may be further included.

The step (a-22) sets a second frequency set including a value obtained by multiplying a black-and-white spectrogram-based basic frequency by positive integers less than or equal to a predetermined value, respectively, at each of the plurality of points in time, and the second frequency set The method may further include setting, as the final fundamental frequency, a value obtained by dividing a frequency having the greatest fundamental frequency suitability among frequencies belonging to by a positive integer multiplied when setting the second frequency set.

In the step (b), the harmonics of the fundamental frequency may have the expected amplitude and phase of a spring having a natural frequency corresponding to the harmonics of the fundamental frequency as the amplitude and phase of the harmonics of the fundamental frequency.

50% to 75% of the harmonics of the fundamental frequency may be included in the entire training data.

A computer-readable recording medium according to an embodiment of the present invention is recorded so that each step of the data augmentation method can be executed by a computer.

According to an embodiment of the present invention, since training data is generated using harmonics of the fundamental frequency obtained by DJ conversion, machine learning performance can be improved even when noise in a new environment is added.

1 is a diagram for explaining how a data augmentation method according to an embodiment of the present invention is used in machine learning.
2 is a flowchart showing each step of a data augmentation method according to an embodiment of the present invention.
3 is a flowchart illustrating an example of the fundamental frequency extraction step of FIG. 2 .
4 is a flowchart illustrating an example of the fundamental frequency extraction step of FIG. 3 .
5 is a flowchart illustrating an example of the fundamental frequency extraction step of FIG. 4 .
6 is a diagram showing an experimental example of the present invention when a sound of harmonics that changes with time is input.
7 is a table showing experimental results for an embodiment of the present invention.
8 is a structural diagram of a data augmentation system according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" means that a stated component, step, operation, and/or element is present in the presence of one or more other components, steps, operations, and/or elements. or do not rule out additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram for explaining how a data augmentation method according to an embodiment of the present invention is used in machine learning.

Referring to FIG. 1 , a dataset must be input in order to generate a machine learning algorithm for speaker recognition or speaker verification. The training dataset of the machine learning system of FIG. 1 is an original training dataset according to the prior art, and may be, for example, a dataset augmented by MUSAN and RIP (Room Impulse Response).

An embodiment of the present invention generates (enhances) data consisting of harmonics of the fundamental frequency extracted through DJ conversion of the original training dataset as a training dataset. Then, the original training dataset and the augmented data according to the embodiment of the present invention are input to the machine learning algorithm. At this time, each of the original training dataset and the augmented data according to the embodiment of the present invention is voice data (eg wav) and a label (a value indicating a speaker in case of speaker recognition, a value indicating whether the speaker is correct in case of speaker verification) can include

Although FIG. 1 shows inputting both the original training dataset and the augmented data according to the embodiment of the present invention to the machine learning algorithm, only the augmented data according to the embodiment of the present invention may be input.

2 is a flowchart showing each step of a data augmentation method according to an embodiment of the present invention. Each step is performed by a computer processor.

First, at least one fundamental frequency of voice data is extracted (S10). The extraction of the fundamental frequency uses a DJ transform modeling the vibrational motion of a plurality of springs, and details will be described later.

Next, harmonics of the fundamental frequency are generated as training data (S20).

The generated training data is used as machine learning training data for speech recognition or speaker recognition.

3 is a flowchart illustrating an example of the fundamental frequency extraction step ( S10 ) of FIG. 2 .

Referring to FIG. 3 , a fundamental frequency extraction method according to an embodiment of the present invention models vibrational motions of a plurality of springs having different natural frequencies for input of voice data, and estimates pure tone amplitude and phase according to the natural frequencies. Calculating and generating a DJ transform spectrogram representing the natural frequency of each of the plurality of springs and the estimated pure tone amplitude according to a plurality of time points (S100); and calculating the fundamental frequency based on a moving average of the estimated pure tone amplitude or a moving standard deviation of the estimated pure tone amplitude for each natural frequency of the DJ conversion spectrogram (S200).

DJ conversion is modeling the vibration motion of a plurality of springs having different natural frequencies, and is intended to represent the characteristics of real sound well by simulating the motion of hair cells in the cochlea of the ear through the vibration motion of the springs. Since frequency can be easily converted into frequency or angular velocity, they are used interchangeably in this specification.

A plurality of springs are set to have different natural frequencies. The natural frequency of the plurality of springs may have a predetermined frequency interval, for example, 1Hz, 2Hz, or 10Hz, in a frequency range corresponding to sound, for example, an audible frequency band of 20Hz to 20kHz.

with spring constant k _i The equation of motion for the external force F(t) of the displacement x _i (t) of an object of mass M fixed at one end of the spring s _i in a parallel position is as follows:

(식1)

(Equation 1)

이고, 감쇠율을 ζ라 할 때 Γ_i는 단위 질량 당 감쇠상수로

이다. 모델에서는 M=1, ζ=0.001을 사용하였는데, 성능 개선을 위해 향후 바뀔 수 있는 값들이다. where ω _oi is the natural resonant angular velocity

, and when the damping rate is ζ, Γ _i is the damping constant per unit mass.

am. In the model, M = 1 and ζ = 0.001 were used, which are values that can be changed in the future for performance improvement.

가 입력된다고 가정하자. 이때, 초기 조건이 정지 상태인 용수철의 운동 방정식의 해 x_i(t)는 다음과 같이 표현된다.External sound with angular velocity ω _ext and constant amplitude F _ext

Assume that is entered. At this time, the solution x _i (t) of the equation of motion of the spring with the initial condition at rest is expressed as follows.

(Equation 2)

인데, 모델에서 ζ는 예를 들어 0.001정도의 매우 작은 값을 사용하면

이 된다. 그리고

와

은 다음과 같다.here

However, in the model, ζ is 0.001, for example, if a very small value is used,

becomes and

and

Is as follows.

(식3)

(Equation 3)

(식4)

(Equation 4)

와

는 다음과 같이 된다.When the angular velocity ω _ext of the external force coincides with the angular velocity ω _0i of the natural frequency of the spring

and

becomes the following

(식5)

(Equation 5)

(식6)

(Equation 6)

인 조건을 만족하는 용수철을 공명 조건의 용수철이라 한다. 이때,

이라고 할 수 있으므로, 용수철의 변위 x_i(t)는 다음과 같이 표현된다.When there is an angular velocity ω _ext of an external sound, the angular velocity ω _0i of the natural frequency of the spring used in DJ conversion is

A spring that satisfies the phosphorus condition is called a spring with resonance conditions. At this time,

, the displacement x _i (t) of the spring is expressed as:

(식7)

(Equation 7)

일 때의 값

은 식5의

일 때의 값

과 거의 같으므로, 식 전개에서 같은 값으로 사용한다. in Equation 7

value when

of Eq. 5

value when

Since it is almost the same as , use the same value in expression expansion.

라 정의하자. 식7을 t=τ_n 인 시점, 즉, 1주기에서 변위 x_i(t)가 최대인 시점에서 관찰하면, x_i(t=τ_n) 의 값은 다음과 같이 간단하게 표현된다.τ _n

let's define Observing Equation 7 at the point of time t=τ _n , that is, the point of maximum displacement x _i (t) in one cycle, the value of x _i (t=τ _n ) is simply expressed as follows.

(식8)

(Equation 8)

의 값에 수렴하게 된다. According to Equation 8, after a sufficient amount of time (n→∞), the displacement x _i (t=τ _n ) of the stabilized state is

converges to the value of

을 구할 수 있다. 과정은 다음과 같다. The convergence value of the displacement x _i (t) in the stable state after a sufficient amount of time has elapsed, after the external sound input starts to come in and before convergence has passed.

can be obtained. The process is as follows.

First, Equation 8 is transformed as follows.

(식9)

(Equation 9)

If the value of n in Equation 9 is changed to n+1, the equation changes to the following.

(식10)

(Equation 10)

By dividing Equation 9 into Equation 10 by each side, the following equation can be obtained.

(식11)

(Equation 11)

일 때, 식 11은 x_i(t=τ_n)과 x_i(t=τ_n+1)의 값을 알면, 시간이 충분히 지난 후의 안정화된 상태의 변위 x_i(t)의 수렴값, 즉 안정상태 예상 진폭

을 추정할 수 있다는 것을 보여준다. 그리고 그 시점에 구한 추정값

과 식5를 이용하여 그 시점의 외부 소리의 세기 F_ext(t)의 크기를 다음과 같이 구할 수 있다.

When the values of x _i (t=τ _n ) and x _i (t=τ _n+1 ) are known, Equation 11 is the convergence value of the displacement x _i (t) of the stable state after a sufficient amount of time has elapsed, that is, Steady-State Expected Amplitude

shows that it is possible to estimate and the estimate obtained at that time

Using Equation 5, the magnitude of the external sound intensity F _ext (t) at that time can be obtained as follows.

(식12)

(Equation 12)

에 기초하여 산출된 외부 소리의 세기 F_ext(t)를 순음 예측 진폭이라 한다.Convergence value of the displacement x _i (t) of the stabilized state in this specification

The intensity of the external sound F _ext (t) calculated based on is referred to as pure tone prediction amplitude.

On the other hand, in order to extract the phase, if the spring motion is viewed as a one-dimensional projection of the uniform circular motion, the phase is as follows.

(식13)

(Equation 13)

Next, assume that a harmonic input consisting of n components whose frequencies are positive integer multiples of the fundamental frequency f ₀ is given. At this time, the set W of the angular velocities of harmonics is as follows.

(식14)

(Equation 14)

Let the elements of the set W be ordered in order from smallest to smallest.

(식15)

(Equation 15)

로 나타낼 수 있다.These harmonics

can be expressed as

Given the harmonic F(t) as an input, the displacement x _i (t) of the spring can be expressed as the sum of displacements of the spring for each angular velocity constituting the frequency set W as follows.

(식16)

(Equation 16)

와

은 다음과 같다.here

and

Is as follows.

(식17)

(Equation 17)

(식18)

(Equation 18)

By observing x _i (t) in the direction of increasing (or decreasing) the size of the angular velocity ω _0i of the natural frequency of the spring, springs that are in resonance with each element of the set of angular velocities W included in harmonics can be found. When observed in an arbitrarily short period of time, the maximum value of displacement x _i (t) of the spring under resonance conditions according to Equations 16, 17, and 18 does not become a resonance condition immediately adjacent to the natural angular velocity of the spring. greater than the maximum value of the displacement x _i (t) of the spring. Therefore, if the spectrogram of the DJ conversion result is created using Equations 11 and 12 for the maximum values for each natural period of the spring displacement x _i (t), the angular velocity value at the point where the maximum value is observed at a specific point in time is the harmonic There is a one-to-one correspondence with the elements of the angular velocity set W.

That is, by modeling the oscillatory motion of the spring, the displacement x _i (t) of the spring expressed by Equations 16 to 18 can be found, and by applying Equations 11 and 12 to the displacement x _i (t) of the spring, various When a sound having a frequency is input, an estimated pure tone amplitude may be calculated. The estimated pure tone amplitude may be the steady state expected amplitude of Equation 11 or the predicted pure tone amplitude of Equation 12. Accordingly, a DJ conversion spectrogram based on the estimated pure tone amplitude can be generated by displaying the estimated pure tone amplitude in the space of the time axis and the frequency axis corresponding to the resonant frequency of the spring.

)인 용수철 s_i의 변위 x_i(t)에

으로 공명 조건이 아닌 각속도 ω_ext,n의 소리 입력으로 변위 x_i(t)의 진폭이 바뀌는 비율은 다음 식에서 추정할 수 있다. In this regard, the displacement x _i (t) corresponding to one maximum value of the spectrogram is greatly affected by the resonance condition among the angular velocity sounds included in the harmonics, but it is also affected by the angular velocity sound, which is not a resonance condition. It can be seen from Eqs. 16, 17, and 18. Given the harmonics, the natural oscillation angular velocity ω _0i is a resonance condition with ω _ext,m , i.e. (

), the displacement x _i (t) of the spring s _i

, the rate at which the amplitude of displacement x _i (t) changes with sound input of angular velocity ω _ext,n, which is not a resonance condition, can be estimated from the following equation.

(식19)

(Equation 19)

이고 공명 조건에서 많이 벗어난 곳에서는

이다. 식19는 그 값 중 큰 값들만 선택하여 비교한 결과이다. F_ext,n 와 F_ext,m 의 값이 크게 차이나지 않으면 ζ=0.001일 때

항의 영향이

보다 훨씬 우세하다는 것을 식19에서 알 수 있다. 공명 조건에 의해 생기는 극댓값의 위치가 바뀔 정도로 고조파의 공명 조건이 아닌 주파수의 영향이 크지는 않다. 따라서 고조파에 포함되어 있는 주파수 위치에서 디제이변환 스펙트로그램에서 극댓값을 관찰할 수 있다. Observing Eqs. 17 and 18, near resonance conditions

and where the resonance conditions are far out of the range,

am. Equation 19 is the result of selecting and comparing only the larger values among the values. If the values of F _ext,n and F _ext,m do not differ significantly, when ζ=0.001

protest impact

It can be seen from Eq. 19 that it is much more dominant than The influence of frequencies other than resonance conditions of harmonics is not large enough to change the position of the maximum value caused by resonance conditions. Therefore, the maximum value can be observed in the DJ transform spectrogram at the frequency position included in the harmonics.

This time, we look at the relationship between the frequency constituting the harmonics and the maximum value of the displacement x _i (t), each of which is a resonance condition. In the DJ transform, the natural frequency f ₀ and the maximum value of the displacement x _i (t) of the spring, which is the resonance condition, are calculated every period of 1/f ₀ . The maximum value of displacement x _i (t) reflects the influence of f _i , which is not the fundamental frequency included in harmonics, but the period 1/f _i of these frequencies becomes a divisor of 1/f ₀ , with a period of 1/f ₀ When calculating, the maximum value of x _i (t) is reflected as a periodic nature. In DJ conversion, f _i, which is not the fundamental frequency, and the maximum value of displacement x _i (t) of the spring, which is a resonance condition, are also calculated every 1/ _fi period. Since the period of the part affected by f ₀ is 1/f ₀ (1/f ₀ > 1/f _i ), if calculated with a period of 1/f _i , the maximum value of displacement x _i (t) will be reflected as a periodic property. can't

Therefore, the maximum value of x _i (t) related to f ₀ does not destroy the periodicity and the amplitude of the value is small, and the maximum value of x _i (t) related to f _i has a large amplitude of vibration due to the destruction of the periodicity. . The amplitude value of the estimated pure-tone amplitude-based spectrogram calculated using the maximum value of x _i (t) through Equations 11 and 12 reflects the characteristics of the maximum value of x _i (t) as it is. Therefore, when calculating the standard deviation of the amplitude values of the spectrogram, the value is small in the part related to f ₀ and the value is large in the part related to f _i .

In summary, when a harmonic is given, a spring that resonates with the fundamental frequency of the harmonic 1) when the amplitude of the spectrogram of its own fundamental frequency is measured, the dispersion of the amplitude over time is small and 2) the maximum value of the amplitude is large. .

Based on these characteristics, the embodiment of the present invention extracts the fundamental frequency based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude for each natural frequency of the DJ transform spectrogram.

4 is a flowchart illustrating an example of the fundamental frequency extraction step 200 of FIG. 3 . Referring to FIG. 4, in the step of extracting the fundamental frequency, the step of calculating the degree of fitness for the fundamental frequency based on the moving average of the estimated pure tone amplitude or the moving standard deviation of the estimated pure tone amplitude in the D-J conversion spectrogram (S210) and each of a plurality of time points. It may include calculating the maximum value of the fitness of the fundamental frequency in , and extracting the fundamental frequency based on the calculated maximum value of the fitness of the fundamental frequency (S220).

Let R(t, f) be the degree of similarity between the natural frequency f of the spring and the fundamental frequency of the input harmonic at time t. As described above, the displacement of the spring that resonates with the fundamental frequency has a small change in value over time compared to the displacement of the spring that resonates with other frequencies constituting harmonics. In addition, the displacement of a spring that resonates with each frequency constituting harmonics has a larger amplitude than that of a spring with an adjacent natural frequency. Using these characteristics, the fundamental frequency fitness R(t, f) of the spring can be calculated by the following Equation 20.

(식20)

(Equation 20)

(식21)

(Equation 21)

(식22)

(Equation 22)

(식23)

(Equation 23)

로 할 수 있다.where N is an integer and ε is a very small value greater than zero. For example, ε at time t is

can be done with

이면

이 되도록 한다. 여기서 β는 작은 값으로 β=10^-12 를 사용할 수 있다.To reduce the influence of small amplitude values on the spectrogram

the other side

let this be Here, β is a small value and β=10 ^-12 can be used.

또는

를 사용할 수도 있다. 즉, 기본주파수 적합도는 추정 순음 진폭을 나타내는 디제이변환 스펙트로그램 S(t, f)의 이동평균 M(t, f)에 비례하거나 이동표준편차 σ(t, f)에 반비례할 수 있다.Depending on the embodiment, instead of (Equation 20)

or

can also be used. That is, the basic frequency fit may be proportional to the moving average M (t, f) of the DJ transform spectrogram S (t, f) representing the estimated pure tone amplitude or inversely proportional to the moving standard deviation σ (t, f).

Next, the fundamental frequency is extracted based on the maximum value of the degree of fitness for the fundamental frequency according to the natural frequency at each time point (S220).

Depending on the embodiment, the fundamental frequency may be extracted as the lowest frequency among frequencies corresponding to the maximum value of the degree of fitness of the fundamental frequency according to the natural frequency at each time point.

5 is a flowchart illustrating an example of the fundamental frequency extraction step ( S220 ) of FIG. 4 .

Referring to FIG. 5, in the basic frequency extraction step (S220), in order to improve accuracy by excluding the influence of noise, the black and white spectrogram generation step (S310), the average black and white spectrogram generation step (S320), and the maximum value of the average black and white spectrogram It may include an extraction step (S330), a candidate fundamental frequency extraction step (S340), a step of setting the basic frequency based on the black and white spectrogram (S350), and a step of setting the final fundamental frequency (S360).

The fundamental frequency extraction step (S220) does not need to include all of the steps of S310 to S360, and may include only some of them according to embodiments.

Depending on the embodiment, the fundamental frequency extraction step (S220) extracts the top N (N is an integer of 2 or more) among the fundamental frequency suitability for each time point, and sets the value corresponding to the natural frequency corresponding to the N to "1". A black and white spectrogram generation step of setting the value to "0" and setting the remaining values to "0" (S310); An average black-and-white spectrogram generating step (S320) of calculating an average of the black-and-white spectrogram for an area of the same size including each point of the black-and-white spectrogram; and extracting a maximum value of the average black-and-white spectrogram at each of the plurality of viewpoints (S330).

의 기본주파수 적합도 R(t,f) 중에서 상위 N개를 추출한다. 상위 N개에 들어가는지 여부를 기준으로 값이 0과 1이 되는 흑백스펙트로그램을 구성한다. 만약 R(t,f) 가 시점 t에서 상위 N개에 들어가면 BW(t,f)=1 그렇지 않으면 BW(t,f)=0 이 되도록 한다.The step of generating the black and white spectrogram (S310) is the time of constructing the DJ conversion spectrogram.

Extract the top N out of the basic frequency fit R(t,f) of . Construct a black and white spectrogram whose values are 0 and 1 based on whether or not it is in the top N. If R(t,f) is in the top N at time t, BW(t,f)=1, otherwise BW(t,f)=0.

라고 하자.In the step of generating the average black and white spectrogram (S320), the average of each point constituting the black and white spectrogram BW(t,f) is obtained in a rectangular region based on itself as shown in the following equation. The result of this configuration is the average black and white spectrogram

let's say

(식24)

(Equation 24)

보다 큰 극댓값들을 추출한다. 여기서 임계값

는 각 시점 t마다 구한

의 극댓값의 최댓값인

에 일정비율 γ (0≤γ≤1.0)을 곱한 값으로 정한다. 예를 들어 γ는 0.2로 설정될 수 있다.In the step of extracting the maximum value of the average black-and-white spectrogram (S330), the given threshold among the maximum values according to the change in natural frequency at each time point t in the average black-and-white spectrogram

Extract larger maxima. Threshold here

is obtained at each time point t

is the maximum of the maxima of

It is set as the value multiplied by a certain ratio γ (0≤γ≤1.0). For example, γ may be set to 0.2.

That is, the extracted maxima simultaneously satisfy the following conditions.

, (식25)

, (Equation 25)

, (식26)

, (Equation 26)

, (0≤γ≤1.0) (식27)

, (0≤γ≤1.0) (Equation 27)

The fundamental frequency extraction step (S220) is a candidate based on the lowest frequency among the natural frequencies corresponding to the maximum value of the average black-and-white spectrogram and the difference between the natural frequencies corresponding to the adjacent maximum values of the average black-and-white spectrogram at each of a plurality of time points. A step of extracting the fundamental frequency (S340) may be further included.

이라 하자. 인접한 주파수들의 간격

을 아래와 같이 계산한다.As a result of sorting the maxima extracted from the average black and white spectrogram at time t in ascending frequency order, the frequency corresponding to the kth maxima

let's say spacing of adjacent frequencies

is calculated as below:

(식28)

(Equation 28)

중

보다 큰 값들을 고르고 그 중 제일 작은 값과

를 비교해서 작은 값을 시점 t에서의 후보 기본 주파수

로 결정한다. 여기서는 음성이나 악기의 음에 존재하는 고조파의 인접한 주파수들의 차이값들 중에서 최솟값이 기본주파수일 가능성이 크다는 사실을 이용하였다.

middle

Choose the larger values and select the smallest value among them.

By comparing , the small value is the candidate fundamental frequency at time t

to decide Here, the fact that among the differences between adjacent frequencies of harmonics present in voice or musical instrument sound, the minimum value is likely to be the fundamental frequency is used.

가 된다.If the amplitudes of all frequencies constituting noise-free harmonics are the same, then for each k

becomes

The basic frequency extraction step (S300) may include a black-and-white spectrogram-based basic frequency setting step (S350). Calculate the time average of the gram-based fundamental frequencies, set a first frequency set including frequencies around values obtained by multiplying the time average by positive integers less than or equal to a predetermined value, and among the frequencies belonging to the first frequency set The method may include setting a value obtained by dividing a frequency having the largest average black-and-white spectrogram by a positive integer multiplied when setting the first frequency set as a basic frequency based on the black-and-white spectrogram.

Here, in order to set the initial value, among the candidate fundamental frequencies for the plurality of viewpoints, a candidate fundamental frequency of a viewpoint having the smallest moving variance of the difference between the candidate fundamental frequencies of adjacent viewpoints is selected. A step of setting the basic frequency based on the black and white spectrogram of the viewpoint may be further included.

를 찾았다고 가정하자. 각 시점 t에 대해 흑백스펙트로그램기반 기본주파수 BF₀(t)를 찾기 위해 첫번째로 특정 시점 t₀ 에서의 흑백스펙트로그램기반 기본주파수 BF₀(t)를 계산한다. 두 번째로 시점 t₀로부터 시간을 증가시키면서 흑백스펙트로그램기반 기본주파수를 계산한다. 세번째로 시점 t₀ 로부터 시간을 감소시키면서 흑백스펙트로그램기반 기본주파수를 계산한다.Candidate fundamental frequencies for each point in time t

Suppose you find To find the fundamental frequency BF ₀ (t) based on the black-and-white spectrogram for each time point t, first, the fundamental frequency BF ₀ (t) based on the black-and-white spectrogram at a specific point in time t ₀ is calculated. Second, the fundamental frequency based on the black-and-white spectrogram is calculated while increasing the time from the point in time t ₀ . Thirdly, while decreasing the time from the point in time t ₀ , the fundamental frequency based on the black and white spectrogram is calculated.

First, the time point t ₀ at which the black-and-white spectrogram-based fundamental frequency is calculated is determined as the point at which the variance of the change over time of the black-and-white spectrogram-based candidate fundamental frequency over time is the smallest. The variance V(t) of the change of the candidate fundamental frequency based on the black-and-white spectrogram at each time point t is calculated by the following formula.

(식29)

(Equation 29)

(식30)

(Equation 30)

(식31)

(Equation 31)

이고 시점 t₀의 기본주파수 BF₀(t₀)는 아래와 같이 후보 기본주파수와 동일한 값으로 확정한다.The point in time t ₀ when V(t) is smallest is

, and the fundamental frequency BF ₀ (t ₀ ) of time t ₀ is determined as the same value as the candidate fundamental frequency as follows.

(식32)

(Equation 32)

In the second step, the fundamental frequency based on the black-and-white spectrogram is calculated while increasing the time from the point in time t ₀ . Assume that the fundamental frequency based on the black-and-white spectrogram is obtained from the time point t ₀ to the time point t _k . Let S(t _k+1 ) be the set of eigenfrequencies near n average frequencies of black-and-white spectrogram-based fundamental frequencies obtained up to the previous point and around positive integer multiple frequencies of this average frequency.

(Equation 33)

here,

(식34)

(Equation 34)

, and may be set to, for example, Δf = 20Hz and i _max =5.

에 속하다고 가정하자. 그러면 시점 t_k+1 에서의 흑백스펙트로그램기반 기본주파수 BF₀(t_k+1)는 아래 식으로 구한다.Among the eigenfrequencies belonging to the set S(t _k+1 ), f _max is the frequency with the largest mean black-and-white spectrogram value, and f _max is the frequency domain.

Suppose we belong to Then, the basic frequency BF ₀ (t _{k +1 ) based on the black-and-white spectrogram at the time point t k+1} is obtained by the following formula.

(식35)

(Equation 35)

Repeat the second step above while increasing k by 1 until t _k+1 is the last time of the given spectrogram.

In the third step, while decreasing the time from time point t ₀ , a process similar to the second step is performed to obtain the black-and-white spectrogram-based fundamental frequency at each time point until t=0.

Next, in the step of extracting the fundamental frequency (S300), at each of the plurality of points in time, a second frequency set including a value obtained by multiplying the black and white spectrogram-based fundamental frequency by positive integers less than or equal to a predetermined value is set, The method may further include setting, as the final fundamental frequency, a value obtained by dividing a frequency having the greatest fundamental frequency suitability among frequencies belonging to the second frequency set by a positive integer multiplied when setting the second frequency set. .

The final fundamental frequency f 0 (t) is to be extracted using the black-and-white spectrogram-based fundamental frequency BF ₀ (t) _at each time point t and the above-described basic frequency fit R(t, f).

라 하자.A set of frequencies near the fundamental frequency BF ₀ (t) and positive integer multiples of BF ₀ (t) based on the black-and-white spectrogram at time t

let's say

(식36)

(Equation 36)

Here, Δf = 20 Hz, i _max = 5 may be set.

에 속하는 주파수 중 기본주파수 적합도 R(t, f)가 제일 큰 주파수가 f_max 이고 f_max 는 주파수 영역

에 속하다고 가정하자. 그러면 시점 t에서의 최종 기본주파수 f₀(t)는 아래 식으로 구한다.set at point t

Among the frequencies belonging to , the frequency with the largest fundamental frequency fit R(t, f) is f _max , and f _max is the frequency domain

Suppose we belong to Then, the final fundamental frequency f ₀ (t) at time t is obtained by the following formula.

(식37)

(Equation 37)

Next, harmonics of the final fundamental frequency f ₀ (t) are generated as training data.

(식38)

(Equation 38)

는 시점 t에서 f₀(t)의 j배 주파수의 진폭이며,

을 만족하고,

는 식(11)로부터 알 수 있다. 또한,

는 시점 t에서 f₀(t)의 j배 주파수의 위상이며,

이고,

는 식(13)으로부터 알 수 있다.here,

is the amplitude of j times the frequency of f ₀ (t) at time t,

satisfies,

can be found from equation (11). also,

is the phase of j times the frequency of f ₀ (t) at time t,

ego,

can be found from equation (13).

For example, if you want to generate audio data (wav data) with a resolution of 1 msec and a sampling frequency of 48000 Hz in DJ conversion, you need to generate 48 audio data at intervals of 1 msec. ) to generate. That is, the interval between the plurality of viewpoints can be 1/48 msec.

6 is a diagram showing an experimental example of the present invention when a sound of harmonics that changes with time is input.

Figure 6 (a) shows the sound input, Figure 6 (b) shows the DJ conversion spectrogram generated using the sound input of Figure 6 (a), Figure 6 (c) shows the After calculating the basic frequency suitability from the DJ conversion spectrogram of (b), the black and white spectrogram generated using the calculated basic frequency suitability is shown, and FIG. 6(d) is the black and white spectrogram of FIG. 6(c) After generating the average black-and-white spectrogram using , the candidate fundamental frequency selected using the maximum value in the frequency direction of the average black-and-white spectrogram and the frequency values at the location is shown, and FIG. The basic frequency based on the black and white spectrogram generated using the candidate fundamental frequency of d) is shown, and FIG. It shows the fundamental frequency, and FIG. 6(g) is a partially enlarged view of FIG. 5(b).

As shown in (a) to (g) of FIG. 6, it can be seen that the final fundamental frequency substantially coincides with the value corresponding to the fundamental frequency of the sound input.

7 is a table showing experimental results for an embodiment of the present invention.

Referring to the upper part of FIG. 7, a data set for speaker identification (VoxCeleb1 Identification Set) is input for 2.58 seconds, data augmented by MUSAN and RIR (Room Impulse Response), respectively, and DJ according to an embodiment of the present invention The augmented data was applied to the thin ResNetSE50 network by generating a transformation spectrogram. At this time, softmax was used as the loss function.

In this experiment, the content of augmented data (harmonics of the fundamental frequency) using the DJ conversion spectrogram is 62.5%. Among the entire augmented data, the content rate of harmonics of the fundamental frequency may be 50% to 75%. If the harmonic content of the fundamental frequency is lower than 50%, the essence of the original data may be damaged, whereas if the content is higher than 75%, the training data may be too close to the original data, reducing the effect of data augmentation. In this experiment, the effect of performance improvement was demonstrated by testing with 62.5%, which is the median value of this content rate range.

According to the prior art, when only data augmented by MUSAN and RIR were used, the accuracy of speaker identification was 94.91%, but when augmented data by DJ conversion spectrogram was included, the accuracy improved to 95.55%.

Referring to the lower part of FIG. 7, a data set for speaker verification (VoxCeleb1 Verification Set) is input for 2.58 seconds, data augmented by MUSAN and RIR (Room Impulse Response), respectively, and DJ conversion according to an embodiment of the present invention The augmented data (62.5% content) was applied to the thin ResNetSE152 network by generating a spectrogram. At this time, angle-proto was used as a loss function.

According to the prior art, when only data augmented by MUSAN and RIR were used, the error rate of speaker verification was 4.279%. can know that

Although various data augmentation methods are used in machine learning, the method of augmenting data by extracting the harmonics of the fundamental frequency of the original data has never been used, and as can be seen from the above experimental results, the effect is very good. can

8 is a structural diagram of a data augmentation system 1 according to an embodiment of the present invention.

Referring to FIG. 8 , the data augmentation system 1 may include a processing device 2 and a memory 3 .

The processing device 2 executes a data augmentation method according to an embodiment of the present invention. The data augmentation method basically includes extracting at least one fundamental frequency of voice data and generating harmonics of the fundamental frequency as training data, and may include the above-described additional steps according to embodiments.

The memory 3 stores data necessary for executing the data augmentation method. For example, the memory 3 stores audio data so that the audio data can be provided to the processing device 2 . In addition, the memory 3 stores the estimated pure tone amplitude, steady state expected amplitude, pure tone predicted amplitude, phase, DJ conversion spectrogram, basic frequency fit, black and white spectrogram, average black and white spectrogram, black and white spectrogram based basic frequency, etc. , the processing device 2 can extract the final fundamental frequency using them.

Although not shown in FIG. 8, the data augmentation system 1 includes a voice input device such as a microphone for receiving voice and converting it into electronic voice data, and a display device such as a monitor for showing data stored in the memory 3. , input/output devices such as a keyboard and a mouse for receiving user input may be additionally included.

Although the present invention has been described in detail through preferred embodiments, the present invention is not limited thereto, and various changes and applications can be made without departing from the technical spirit of the present invention. self-explanatory for technicians Therefore, the true scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (14)

Each step is performed by a computer processor and is a data augmentation method for generating machine learning training data for speech recognition or speaker recognition,
(a) extracting at least one fundamental frequency of voice data; and
(b) generating harmonics of the fundamental frequency as training data;
A data augmentation method comprising a.
According to claim 1,
In step (a),
(a-1) With respect to the input of voice data, the vibration motion of a plurality of springs having different natural frequencies is modeled, respectively, to calculate the estimated pure tone amplitude and phase according to the natural frequencies, and the natural frequency of each of the plurality of springs is calculated. generating a DJ-converted spectrogram representing the estimated pure tone amplitude according to a corresponding frequency and a plurality of time points; and
(a-2) extracting the fundamental frequency based on a moving average of the estimated pure tone amplitude or a moving standard deviation of the estimated pure tone amplitude for each natural frequency of the DJ transform spectrogram;
A data augmentation method comprising a.

According to claim 2,
In the step (a-1),
estimating a steady-state expected amplitude, which is a convergence value of amplitudes of each of the plurality of springs in a stable state, based on amplitudes of two time points of a unique one-cycle interval of each of the plurality of springs; and
Calculating the estimated pure tone amplitude based on the predicted pure tone amplitude, which is the amplitude of the input sound estimated based on the expected steady state amplitude.
A data augmentation method comprising a.
According to claim 3,
The estimated pure tone amplitude is the steady state expected amplitude or the predicted pure tone amplitude.
According to claim 2,
In the step (a-2),
(a-21) calculating a degree of fitness for the fundamental frequency based on a moving average of the estimated pure-tone amplitudes or a moving standard deviation of the estimated pure-tone amplitudes in the DJ transform spectrogram; and
(a-22) calculating the maximum value of the fitness for the fundamental frequency at each of the plurality of points in time, and extracting the fundamental frequency based on the calculated maximum value of the fitness for the fundamental frequency;
Data augmentation method comprising a.

According to claim 5,
The data augmentation method, characterized in that the degree of fitness for the fundamental frequency is proportional to a moving average of the estimated pure-tone amplitude or inversely proportional to a moving standard deviation of the estimated pure-tone amplitude.
According to claim 5,
In the step (a-22),
At each of the plurality of points in time, the top N (N is an integer of 2 or more) of the fundamental frequency conformity are extracted, the values corresponding to the natural frequencies corresponding to the N are set to "1", and the remaining values are set to "0". Black and white spectrogram generation step set to ";
an average black-and-white spectrogram generating step of calculating an average of the black-and-white spectrogram for an area of the same size including each point of the black-and-white spectrogram; and
Extracting the maximum value of the average black-and-white spectrogram at each of the plurality of time points
Data augmentation method comprising a.
According to claim 7,
In the step (a-22),
Extracting a candidate fundamental frequency based on the difference between natural frequencies corresponding to adjacent maxima of the average black-and-white spectrogram and the lowest frequency among the natural frequencies corresponding to the maximum of the average black-and-white spectrogram at each of the plurality of points in time. ;
Data augmentation method characterized in that it further comprises.
According to claim 8,
In the step (a-22),
A first frequency set including frequencies near a value obtained by calculating a time average of black-and-white spectrogram-based basic frequencies set for a predetermined time interval including adjacent time points, and multiplying the time average by positive integers less than or equal to a predetermined value, respectively. Set a value obtained by dividing a frequency having the largest average black-and-white spectrogram among frequencies belonging to the first frequency set by a positive integer multiplied when setting the first frequency set as the black-and-white spectrogram-based fundamental frequency. step to do
Data augmentation method characterized in that it further comprises.
According to claim 9,
In the step (a-22),
Among the candidate fundamental frequencies for the plurality of viewpoints, a candidate fundamental frequency at a time point in which the movement variance of the difference between the candidate fundamental frequencies of adjacent viewpoints is the smallest is determined as the black-and-white spectrogram-based fundamental frequency at the viewpoint in which the movement variance is the smallest. setting up;
Data augmentation method characterized in that it further comprises.
According to claim 9,
In the step (a-22),
At each of the plurality of points in time, a second frequency set including a value obtained by multiplying a black and white spectrogram-based fundamental frequency by positive integers less than or equal to a predetermined value is set, and among the frequencies belonging to the second frequency set, the basic frequency conformity Setting a value obtained by dividing the largest frequency by a positive integer multiplied when setting the second frequency set as the final fundamental frequency.
Data augmentation method characterized in that it further comprises.
According to claim 3,
In the step (b), the harmonics of the fundamental frequency have the expected amplitude and phase of a spring having a natural frequency corresponding to the harmonics of the fundamental frequency as the amplitude and phase of the harmonics of the fundamental frequency. data augmentation method.
According to claim 1,
Data augmentation method, characterized in that 50% to 75% of the harmonics of the fundamental frequency are contained in the entire training data.

A computer-readable recording medium on which each step of claim 1 is recorded so that it can be executed by a computer.

Images