KR20210107943A - Method and Device for Detecting Sound Source - Google Patents

Abstract

Disclosed are a method and a device for processing audio data. According to an aspect of the present invention, provided is a method for processing audio data comprising the steps of: converting input audio data and reference audio data respectively into an input spectrogram and a reference spectrogram; normalizing the reference spectrogram and detecting a target spectrogram, which is a region with the highest similarity with the normalized reference spectrogram in the input spectrogram; generating a power ratio vector, which is the ratio of the power of the target spectrogram to the reference spectrogram; adjusting the size of the reference spectrogram for each time component using the power ratio vector; generating a difference spectrogram based on the target spectrogram and the adjusted reference spectrogram; and generating output audio data based on the difference spectrogram. Therefore, the reference audio data can be detected accurately and deleted from synthesized audio data.

Description

Audio data processing apparatus and method {Method and Device for Detecting Sound Source}

Embodiments of the present invention relate to an audio data processing apparatus and method for removing specific audio data from audio data mixed with various audio data and adding other audio data.

The content described in this section merely provides background information on the present invention and does not constitute the prior art.

When a video is produced, a title, logo, caption, background music (BGM), sound effects, etc. are added to the recorded video after shooting using a plurality of cameras. Original images, subtitles, titles, logos, background music, and sound effects used for mastering are not stored separately due to storage space issues. For this reason, there is a problem in that it is not easy to change the background music or change the voice included in the mastered audio data. In addition, in order to export the mastered video overseas, depending on the country, it is necessary to pay a copyright fee due to the license of the background music (BGM) used in the video, or export may not be possible due to copyright issues.

Therefore, it is necessary to study a technique for removing only specific audio data from the mastered audio data.

To this end, an audio data processing method using a neural network is being studied. There is a technique for inputting synthetic audio data into a trained neural network and acquiring a plurality of separated audio data. For example, if synthetic audio data in which a human voice and background music are synthesized is input to a trained neural network, audio data in which sounds other than a specific human voice are removed can be obtained. Here, the neural network may be one of various neural networks, such as a convolution neural network (CNN), a recurrent neural network (RNN), an auto encoder, a U-Net, a long-short term memory (LSTM), and a generative and adversarial network (GAN).

In addition, a method for separating synthesized audio data in which several audio data is synthesized is being studied, which is divided into a method of separating synthesized audio data in a time-frequency domain and a method of separating synthesized audio data in a time domain. In order to increase separation accuracy, a method of separating synthesized audio data in a time-frequency domain is widely used among the two methods. Specifically, synthesized by converting the synthesized audio data in the time domain and the reference audio data to be removed into the time-frequency domain, removing the reference audio data from the synthesized data, and converting the audio data from which the reference audio data is removed back into the time domain Reference audio data may be removed from the audio data.

However, in order to remove the reference audio data from the synthesized audio data, it is necessary to accurately detect a section in which the reference audio data is used in the synthesized audio data and to remove only a portion having a high degree of relevance to the reference audio data. In addition, in order to remove the reference audio data from the synthesized audio data, it is necessary to adjust the size of the reference audio data in consideration of the size of the synthesized audio data. For example, when fading out (change in volume over time) is applied to a section to be removed from synthesized audio data, a change in volume should be considered in processing the audio data.

When the mastered synthesized audio data detects a change in the volume of the background music used in the synthesized audio data due to voice and noise, in the prior art, a person directly listens to the change in the volume of the background music and determines the background music section. There is a problem in that it is difficult to accurately measure the volume of the background music.

Therefore, it is necessary to study a method for removing reference audio data from synthetic audio data in consideration of the section and size of reference audio data to be removed from synthetic audio data.

Embodiments of the present invention remove the reference audio data from the synthesized audio data while minimizing the influence on other audio data, but include the section including the reference audio data and the size of the reference audio data (including the amount of change in volume over time) An object of the present invention is to provide an audio data processing apparatus and method for accurately detecting and removing reference audio data from synthesized audio data in consideration of the present invention.

Another aspect of the present invention is to provide an audio data processing apparatus and method for using information used to detect and remove reference audio data from synthetic audio data to synthesize other audio data into synthetic audio data. have.

According to one aspect of the present invention, in an audio data processing method for removing or replacing reference audio data from input audio data, input audio data expressed in a time domain and reference audio data expressed in a time-frequency domain are input spectrograms expressed in a time-frequency domain. (spectrogram) and the process of converting to a reference spectrogram, respectively; normalizing the reference spectrogram; detecting a target spectrogram that is a region having the highest similarity with a normalized reference spectrogram in the input spectrogram; generating a power ratio vector that is a ratio of power for each time component of the target spectrogram to power for each time component of the reference spectrogram; adjusting the size of each time component of the reference spectrogram by using the power ratio vector to emphasize a portion of the reference audio data used for the input audio data; generating a differential spectrogram based on the target spectrogram and the adjusted reference spectrogram; and generating output audio data based on the differential spectrogram.

According to another aspect of this embodiment, there is provided an audio data processing apparatus for removing or replacing reference audio data from input audio data, comprising: at least one memory for storing instructions; and at least one processor for processing audio data by executing at least one instruction stored in the memory, wherein the at least one memory provides input audio data expressed in a time domain and a reference through the at least one processor. The audio data is converted into an input spectrogram and a reference spectrogram expressed in a time-frequency domain, respectively, the reference spectrogram is normalized, and the target spectrogram is an area having the highest similarity to the normalized reference spectrogram in the input spectrogram. To detect a gram, generate a power ratio vector that is a ratio of power for each temporal component of the target spectrogram to power for each temporal component of the reference spectrogram, and to emphasize a portion of the reference audio data used for the input audio data , adjusts the size of each time component of the reference spectrogram using the power ratio vector, generates a differential spectrogram based on the target spectrogram and the adjusted reference spectrogram, and outputs audio data based on the differential spectrogram An audio data processing device configured to generate

As described above, according to an embodiment of the present invention, the reference audio data is removed from the synthesized audio data, but considering the section including the reference audio data and the size of the reference audio data (including the amount of change in volume over time) It is possible to accurately detect the reference audio data from the synthesized audio data and remove the reference audio data while minimizing the influence on other audio data.

As described above, according to another embodiment of the present invention, other audio data can be optimally synthesized into the synthesized audio data using information used to detect and remove the reference audio data from the synthesized audio data.

FIG. 1 is a diagram illustrating a process of measuring a degree of similarity between two pieces of data when all of reference audio data is included in input audio data.
FIG. 2 is a diagram illustrating a process of measuring a similarity between two data when input audio data includes only a part of reference audio data.
3 is a diagram illustrating an audio data processing method according to an embodiment of the present invention.
4 is a flowchart illustrating a process of generating a power ratio vector according to an embodiment of the present invention.
5 is a flowchart illustrating an audio data processing process according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, terms such as '~ unit' and 'module' described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

Hereinafter, audio data refers to data obtained by converting the amplitude of a sound wave into a vector according to time, and a spectrogram refers to two-dimensional data obtained by converting audio data into a time-frequency domain. The horizontal axis of the spectrogram means time, and the vertical axis means frequency. An element in a spectrogram is the same word as a bin.

1A and 1B are diagrams for explaining a process of measuring the similarity between two data when all of the reference audio data is included in the input audio data according to an embodiment of the present invention.

Referring to FIGS. 1A and 1B , an input spectrogram 100 , a target spectrogram 102 , and a reference spectrogram 110 are illustrated.

The input spectrogram 100 is two-dimensional data obtained by converting input audio data in which reference audio data and other audio data are mixed into a time-frequency domain.

The reference spectrogram 110 is data obtained by converting the reference audio data necessary for removing a portion similar to the reference audio data from the input audio data into a time-frequency domain. According to an embodiment of the present invention, the reference spectrogram 110 may be normalized.

The target spectrogram 102 is an area in the input spectrogram 100 with the highest similarity to the reference spectrogram 110 .

The audio data processing apparatus (not shown) matches the first time component of the input spectrogram 100 with the first time component of the reference spectrogram 110 to detect the target spectrogram 102 , and the input spectrogram A first similarity (similarity_1) is detected through the sum of (100) and the product of each element of the reference spectrogram 110 .

The audio data processing apparatus moves the reference spectrogram 110 in units of time components, and calculates a similarity between the input spectrogram 100 corresponding to the moved reference spectrogram 110 and the reference spectrogram 110 .

Referring to FIG. 1B , the position of the target spectrogram 102 , which is a section having the highest similarity between the input spectrogram 100 and the reference spectrogram 110 , is shown. When the similarity between the target spectrogram 102 and the reference spectrogram is referred to as a second degree of similarity (similarity_2), the second degree of similarity is greater than the first degree of similarity.

In order to obtain the similarity according to an embodiment of the present invention, the audio data processing apparatus may use at least one of Equation 1, Equation 2, and Equation 3.

는 기준 스펙트로그램(110)의 각 시간 성분에 대한 L2-norm을 의미하는 벡터로서, 수학식 4를 통해 계산된다.In Equations 1, 2 and 3, S is the degree of similarity, i is each time component index of the input spectrogram 100, t is the total time length of the reference spectrogram 110, and f is each time of the spectrogram. The total frequency length of the component, isound means the input spectrogram 100, and rsound means the reference spectrogram 110.

is a vector indicating L2-norm for each time component of the reference spectrogram 110, and is calculated through Equation (4).

2(a) and 2(b) are diagrams for explaining a process of measuring a similarity between two data when only a part of reference audio data is included in input audio data according to an embodiment of the present invention.

Referring to FIGS. 2A and 2B , a zero-padded input spectrogram 200 , a reference spectrogram 210 , and a target spectrogram 202 are illustrated.

When the input audio data includes only a part of the reference audio data, that is, when the input spectrogram includes only a partial section of the reference spectrogram, the audio data processing device (not shown) performs zero padding on the input spectrogram to increase the accuracy of the similarity. carry out

The audio data processing apparatus performs zero padding on the front and rear ends of the input spectrogram for the length of time of the reference spectrogram 210 . In this case, the frequency length of the zero pad, the frequency length of the input spectrogram, and the frequency length of the reference spectrogram 210 are the same.

The audio data processing apparatus determines, as the target spectrogram 202 , a region in which the sum of the product of each element with the reference spectrogram 210 in the zero-padded input spectrogram 200 is largest.

As shown in FIG. 2B , when the entire section of the reference spectrogram 210 is not included in the zero-padded input spectrogram 200 , the audio data processing apparatus performs the zero-padded input spectrogram 200 and the reference spectrogram By measuring the degree of similarity between (210), the accuracy of the degree of similarity can be improved.

3 is a diagram illustrating an audio data processing method according to an embodiment of the present invention.

Referring to FIG. 3 , input audio data 300 , reference audio data 310 , input spectrogram 302 , target spectrogram 304 , target power vector 306 , adjusted target power vector 308 , Reference audio data 310 , reference spectrogram 314 , reference power vector 318 , power ratio vector 320 , adjusted reference spectrogram 330 , and differential spectrogram 340 are shown.

Hereinafter, it will be described that the time component length of the input audio data 300, the input spectrogram 302, and the differential spectrogram 340 is 5, and the target spectrogram 304, the target power vector 306, and the adjusted It is assumed that the time component length of the target power vector 308 , the reference audio data 310 , the reference spectrogram 314 , the reference power vector 318 , the power ratio vector 320 and the adjusted reference spectrogram 330 is 3 Although described, this is only an example, and the audio data processing method may be applied to data of various lengths. In addition, in FIG. 3 , a thick line indicates a part or all of the reference audio data used for the input audio data 300 , and a dotted line indicates the whole of the input spectrogram 302 . Meanwhile, the time component means a set of elements corresponding to one time interval in the spectrogram, and the value of each element means the size of the frequency component.

Referring back to FIG. 3 , the audio data processing apparatus (not shown) removes or replaces the part including the reference audio data 310 from the input audio data 300, so that the input audio data 300 expressed in the time domain ) and the reference audio data 310 are converted into an input spectrogram 302 and a reference spectrogram 314 expressed in a time-frequency domain. According to an embodiment of the present invention, various methods such as Fast Fourier Transform (FFT), Short Time Fourier Transform (STFT), Chroma-STFT, Chroma-CQT and Chroma-CQT (Constant-Q Transform) may be used.

The audio data processing apparatus detects the target spectrogram 304 , which is an area having the highest similarity with the reference spectrogram 314 in the input spectrogram 302 . According to an embodiment of the present invention, the audio data processing apparatus may obtain a similarity between two audio data in the time domain and then use it to detect the target spectrogram 304 .

According to an embodiment of the present invention, the audio data processing apparatus may normalize the reference spectrogram 314 . Normalization of the reference spectrogram 314 is performed by dividing each temporal component of the reference spectrogram 314 by the L2-norm vector using the L2-norm vector calculated by Equation (4). The audio data processing apparatus may detect the target spectrogram 304 , which is a region having the highest similarity with the normalized reference spectrogram in the input spectrogram 302 . In this case, the target spectrogram 304 means an area in which the sum of the product of each element with the reference spectrogram 314 in the zero-padded input spectrogram is the largest.

The audio data processing apparatus generates a power ratio vector 320 that is a ratio of power for each time component of the target spectrogram 304 to the power for each time component of the reference spectrogram 314 . A process in which the audio data processing apparatus generates the power ratio vector 320 will be described in detail with reference to FIG. 4 . In the power ratio vector 320 , the adjusted target power vector 308 , and the adjusted reference spectrogram 330 , an element having a value of 0 is a portion with low relevance to the reference audio data 310 , and is a boundary line of the portion to be removed. becomes this A description of this will also be described with reference to FIG. 4 .

The power ratio vector 320 is used to remove as much as the thick line portion of the reference spectrogram 314 from the thick line portion of the target spectrogram 302 . The power ratio vector 320 not only includes the volume size of the reference audio data 310 used for the input audio data 300 , but also includes information on the amount of change in volume. Due to this, even if the volume of the reference audio data 310 in the input audio data 300 is changed like fading out of the background music, only the reference audio data 310 can be removed without affecting other audio. have. In addition, the power ratio vector 320 may be used to adjust additional audio data to be added to the input audio data 300 . For example, when the reference audio data 310 is synthesized with the input audio data 300 by fading out, the fading-out effect of the reference audio data 310 may be applied equally to the additional audio data. Meanwhile, a normalized reference spectrogram may be used to generate the power ratio vector 320 according to an embodiment of the present invention.

The audio data processing apparatus uses the power ratio vector 320 to emphasize the portion (thick line portion) used for the input audio data 300 among the reference audio data 310 for each time component of the reference spectrogram 314 . Resize. The adjusted reference spectrogram 330 is data for removing a portion having a high degree of relevance to the reference spectrogram 314 from among the target spectrogram 304 . Specifically, the audio data processing apparatus generates the adjusted reference spectrogram 330 through the product of the power ratio vector 320 and the reference spectrogram 314 for each time component. In other words, each time component of the adjusted reference spectrogram 330 is a value obtained by multiplying each time component of the reference spectrogram 314 by each time component of the power ratio vector 320 .

The audio data processing apparatus generates the differential spectrogram 340 based on the target spectrogram 304 and the adjusted reference spectrogram 330 . The differential spectrogram 340 according to an embodiment of the present invention may be data obtained by subtracting the adjusted reference spectrogram 330 from the target spectrogram 304 .

The audio data processing apparatus generates output audio data based on the differential spectrogram 340 . As a result, the output audio data is data in which a portion corresponding to the reference audio data 310 is removed from the input audio data 300 .

4 is a flowchart illustrating a process of generating a power ratio vector according to an embodiment of the present invention.

3 and 4 , the audio data processing apparatus adjusts the target spectrogram 304 through elementwise product of the target spectrogram 304 and the normalized reference spectrogram ( S400 ). Through this process, noise having a low relevance to the normalized reference spectrogram in the target spectrogram 304 is reduced. That is, the similarity between the target spectrogram 304 and the reference spectrogram 314 increases.

The audio data processing apparatus separately generates the adjusted target spectrogram and the target power vector 306 and the reference power vector 318 indicating power for each time component with respect to the reference spectrogram (S402). The target power vector 306 and the reference power vector 318 are derived from the target spectrogram 304 and the reference spectrogram 314 through at least one of Equations 5 and 6.

In Equations 5 and 6, ivolume denotes a target power vector 306 and rvolume denotes a reference power vector 318 .

In addition, the audio data processing apparatus according to an embodiment of the present invention detects one or more sequences having a value smaller than a threshold value in the target power vector 306 and having the same value as at least one of adjacent elements, One sequence with the longest length is determined among one or more sequences, and values of elements excluding the one sequence and an element having a value greater than a threshold value in the target power vector 306 may be substituted with 0 (S404, S404, S406, S408). A portion having a value of 0 is a portion having a low relevance to the input audio data 300 among the reference audio data 310 .

When there are a plurality of sequences according to another embodiment of the present invention, the audio data processing apparatus calculates the similarity between the spectrogram corresponding to the sequence and the input spectrogram, and then values elements except for the sequence corresponding to the spectrogram with the highest similarity. can be substituted with 0.

According to another embodiment of the present invention, the audio data processing apparatus may substitute 0 for an element having a value smaller than a threshold value among elements of the target power vector 306 .

The audio data processing apparatus determines the ratio of the reference power vector 318 to the substituted target power vector 308 as the power ratio vector 320 (S410). Specifically, each element of the power ratio vector 320 is a ratio of the magnitude of each element of the reference power vector 318 to each element of the target power vector 308 .

When the audio data processing apparatus adjusts the reference spectrogram 314 using the power ratio vector 320 , it is possible to accurately detect and remove a thick line portion of the reference audio data 310 used for the input audio data 300 . . Also, when the audio data processing apparatus intends to add other audio data to the output audio data, if the power ratio vector 320 is used, the addition of the other audio data to the output audio data may be optimized.

5 is a flowchart illustrating an audio data processing process according to an embodiment of the present invention.

Referring to FIG. 5 , the audio data processing apparatus converts input audio data and reference audio data into an input spectrogram and a reference spectrogram, respectively ( S500 ).

The audio data processing apparatus normalizes the reference spectrogram (S502).

The audio data processing apparatus detects a target spectrogram, which is an area having the highest similarity with the normalized reference spectrogram in the input spectrogram ( S504 ).

The audio data processing apparatus generates a power ratio vector that is a ratio of the power for each time component of the target spectrogram to the power for each time component of the reference spectrogram ( S506 ).

The audio data processing apparatus adjusts the size of each time component of the reference spectrogram by using the power ratio vector in order to emphasize the portion used for the input audio data among the reference audio data (S508).

The audio data processing apparatus generates a differential spectrogram based on the target spectrogram and the adjusted reference spectrogram (S510).

The audio data processing apparatus generates output audio data based on the differential spectrogram (S512).

Hereinafter, a process in which the audio data processing apparatus according to an embodiment of the present invention adds other audio data to input audio data by using a power ratio vector will be described.

The audio data processing apparatus according to an embodiment of the present invention may use the power ratio vector to add other audio data to the input audio data from which the reference audio data has been removed.

The audio data processing apparatus converts the additional audio data to be added to the input audio data into an additional spectrogram. The audio data processing apparatus adjusts the additional spectrogram using the power ratio vector. This is a process for naturally synthesizing additional audio data into input audio data. The process of adjusting the additional spectrogram is similar to the process of generating the adjusted reference spectrogram 330 in FIG. 3 . That is, the adjusted additional spectrogram is generated through the product of the additional spectrogram and the power ratio vector for each time component.

The audio data processing apparatus generates the sum spectrogram based on the differential spectrogram and the additional spectrogram.

The audio data processing apparatus converts the summed spectrogram into output audio data. In other words, the output audio data becomes data obtained by synthesizing additional audio data in the removed portion after the reference audio data is removed from the input audio data.

Meanwhile, the audio data processing apparatus according to an embodiment of the present invention may pre-process or post-process audio data using a trained neural network. Specifically, noise of the output audio data may be reduced by acquiring the input audio data after inputting the synthesized audio data to the trained neural network, or by inputting the output audio data from which the reference audio data is removed to the trained neural network.

Although it is described that processes S400 to S512 are sequentially executed in FIGS. 4 and 5 , this is merely illustrative of the technical idea of an embodiment of the present invention. In other words, those of ordinary skill in the art to which an embodiment of the present invention pertain may change the order described in FIGS. Since it will be possible to apply various modifications and variations to parallel execution of one or more processes in S512, FIGS. 4 and 5 are not limited to a time-series order.

Meanwhile, the processes shown in FIGS. 4 and 5 can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. That is, the computer-readable recording medium may be a non-transitory medium such as ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device, and also carrier wave (for example, , transmission over the Internet) and may further include a transitory medium such as a data transmission medium. In addition, the computer-readable recording medium is distributed in a network-connected computer system so that the computer-readable code can be stored and executed in a distributed manner.

An audio data processing apparatus according to an embodiment of the present invention includes at least one memory for storing instructions and at least one processor for processing audio data by executing at least one instruction stored in the memory, The at least one memory may store instructions set to perform the operation process of FIGS. 1 to 5 through the at least one processor.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be interpreted as being included in the scope of the present embodiment.

302: input spectrogram 304: target spectrogram
314: reference spectrogram 320: power ratio vector
340: differential spectrogram

Claims (7)

A method for processing audio data, comprising:
converting input audio data and reference audio data expressed in a time domain into an input spectrogram and a reference spectrogram expressed in a time-frequency domain, respectively;
normalizing the reference spectrogram;
detecting a target spectrogram that is a region having the highest similarity with a normalized reference spectrogram in the input spectrogram;
generating a power ratio vector that is a ratio of power for each time component of the target spectrogram to power for each time component of the reference spectrogram;
adjusting the size of each time component of the reference spectrogram using the power ratio vector;
generating a differential spectrogram based on the target spectrogram and the adjusted reference spectrogram; and
generating output audio data based on the differential spectrogram;
Audio data processing method comprising a. According to claim 1,
The process of detecting the target spectrogram,
performing zero padding on the front and rear ends of the input spectrogram for the length of time of the reference spectrogram; and
determining, as the target spectrogram, a region in which the sum of the products of each element with the normalized reference spectrogram in the zero-padded input spectrogram is greatest;
Audio data processing method comprising a. According to claim 1,
The process of generating the power ratio vector is,
adjusting the target spectrogram through elementwise product of the target spectrogram and the normalized reference spectrogram;
separately generating a target power vector and a reference power vector indicating the adjusted target spectrogram and power for each time component with respect to the reference spectrogram;
a process of substituting 0 for a value of an element having a value smaller than a threshold value among elements of a target power vector; and
determining a ratio of the reference power vector to the substituted target power vector as the power ratio vector;
Audio data processing method comprising a. According to claim 1,
The process of generating the power ratio vector is,
adjusting the target spectrogram through elementwise product of the target spectrogram and the normalized reference spectrogram;
separately generating a target power vector and a reference power vector indicating power for each time component with respect to the adjusted target spectrogram and the reference spectrogram;
detecting one or more sequences having a value smaller than a threshold value in a target power vector and having the same value as at least one of adjacent elements;
determining one sequence having the longest length among the one or more sequences;
substituting 0 for values of elements other than the one sequence and an element having a value greater than the threshold value in the target power vector; and
determining a ratio of the reference power vector to the substituted target power vector as the power ratio vector;
Audio data processing method comprising a. 5. The method of any one of claims 3 or 4,
The target power vector is a vector derived from the sum of products of the target spectrogram and the reference spectrogram for each time component,
The reference power vector is an audio data processing method in which the reference spectrogram is a vector derived from a sum of products of each time component with the reference spectrogram. According to claim 1,
The process of generating the output audio data includes:
converting additional audio data to be added to the input audio data into an additional spectrogram;
adjusting the additional spectrogram using the power ratio vector;
generating a summed spectrogram based on the differential spectrogram and the additional spectrogram; and
converting the summed spectrogram into the output audio data;
Audio data processing method comprising a. A computer-readable recording medium in which a program for executing the method of any one of claims 1 to 6 in a computer is recorded.

Images