KR20220109299A - Method for Encoding and Decoding an Audio Signal Using a Neural Network Model, and an Encoder and Decoder Performing the Method - Google Patents

Abstract

Disclosed are a method for encoding and decoding an audio signal using a learning model and an encoder and decoder performing the same. According to one embodiment of the present invention, the method for encoding and decoding an audio signal using the learning model may include: a step of extracting pitch information on the audio signal; a step of determining an expansion factor of a first expandable neural network block extracting a feature map from the audio signal based on the pitch information; a step of determining a receptive field of the first expandable neural network block according to the expansion factor; a step of generating a first feature map of the audio signal by using the first expandable neural network block; a step of determining a second feature map by inputting the first feature map to a second expandable neural network block processing a feature map; and a step of converting the second feature map and the pitch information into a bitstream. The present invention can effectively remove long-term redundancy in the audio signal.

Description

Method for Encoding and Decoding an Audio Signal Using a Neural Network Model, and an Encoder and Decoder Performing the Method}

The present invention relates to a method for encoding and decoding an audio signal using a neural network model, and an encoder and decoder for performing the method, and more specifically, to an audio signal embedded in an audio signal using a neural network model utilizing pitch information of the audio signal. It relates to a technique for encoding and decoding that removes redundancy.

Recently, as the technology for artificial intelligence develops, it is applied to various fields such as processing of voice, audio signal, language, and image signal, and research on it is actively conducted. As a representative example, a technique for extracting features of an audio signal using an autoencoder based on deep learning and restoring the audio signal based on the extracted features is used.

However, in restoring an audio signal, when using a conventional artificial intelligence model, the problem of increased computational complexity, short-term redundancy and long-term redundancy inherent in audio signals, etc. There may be inefficient problems in the removal, so a technique for solving these problems is required.

The present invention provides a method for effectively removing long-term redundancy inherent in an audio signal in a process of encoding and decoding an audio signal by variably determining a dilation factor of a neural network model using pitch information of the audio signal, and provide the device.

In addition, the present invention provides a method and apparatus capable of improving the quality of a reconstructed audio signal and reducing computational complexity by determining an extension factor of a neural network model using pitch information of the audio signal.

A method of encoding an audio signal using a neural network model according to an embodiment of the present invention includes extracting pitch information for the audio signal; determining an extension factor for a receptive field of a first scalable neural network block extracting a feature map from the audio signal based on the pitch information; generating a first feature map of the audio signal by using the first extended neural network block; determining a second feature map by inputting the first feature map to a second extended neural network block that processes the first feature map; Quantizing each of the second feature map and the pitch information may include converting the second feature map and the pitch information into a bitstream.

The generating of the first feature map may include converting the number of channels of the audio signal and inputting it to the first extended neural network block to generate the first feature map and determining the second feature map, The method may further include converting the determined number of channels of the second feature map.

The determining of the second feature map includes down-sampling to reduce a dimension of the first feature map, and inputting the down-sampled first feature map to the second scalable neural network block to obtain the second feature map. A feature map can be determined.

The expansion factor of the first extended neural network block may be determined by approximating an accommodation area of the first extended neural network block with the pitch information.

The extension factor of the second extended neural network block may be fixed to a predetermined value, and the accommodation area of the second extended neural network block may be determined by the fixed extension factor of the second extended neural network block.

The method may further include quantizing the second feature map and the pitch information, respectively, and converting to the bitstream may include converting the quantized second feature map and the pitch information into a multiplexed bitstream.

A method for decoding an audio signal using a neural network model according to an embodiment of the present invention extracts a second feature map for the audio signal and pitch information of the audio signal from a bitstream received from an encoder. to do; reconstructing a first feature map by inputting the second feature map to a second extended neural network block; determining an expansion factor of a first scalable neural network block based on the pitch information; and reconstructing an audio signal from the first feature map using the first extended neural network block.

The step of restoring the first feature map may include: converting the number of channels of the second feature map and inputting it to the second extended neural network block to restore the first feature map and restoring the audio signal; The method may further include converting the number of channels of the restored audio signal to be the same as the number of channels of the original input signal.

The restoring of the audio signal includes up-sampling to extend a dimension of the first feature map, and inputting the up-sampled first feature map to the first extended neural network block to determine the audio signal. can

The expansion factor of the first extended neural network block may be determined by approximating an accommodation area of the first extended neural network block to the pitch information.

The expansion factor of the second extended neural network block may be fixed to a predetermined value, and the accommodation area of the second extended neural network block may be determined by the expansion factor of the second extended neural network block.

The extracting of the second feature map and the pitch information of the audio signal may further include inverse-quantizing the second feature map and the pitch information, respectively.

In an encoder for performing a method of encoding an audio signal according to an embodiment of the present invention, the encoder includes a processor, the processor extracting pitch information for the audio signal, and adding the pitch information to the audio signal. determining an expansion factor of a first extended neural network block for extracting a feature map from the audio signal based on the first extended neural network block, generating a first feature map of the audio signal using the first extended neural network block, A second feature map may be determined by inputting the first feature map to a second extended neural network block that processes the map, and the second feature map and the pitch information may be converted into a bitstream through quantization, respectively.

The processor may down-sample to reduce a dimension of the first feature map, and input the down-sampled first feature map to the second scalable neural network block to determine the second feature map.

The expansion factor of the first extended neural network block may be determined by approximating an accommodation area of the first extended neural network block with the pitch information.

The expansion factor of the second extended neural network block may be fixed to a predetermined value, and the accommodation area of the second extended neural network block may be determined by the expansion factor of the second extended neural network block.

In a decoder for performing the method of encoding an audio signal according to an embodiment of the present invention, the decoder includes a processor, wherein the processor includes a second feature map (feature map) for the audio signal in a bitstream received from the encoder. map) and the audio signal pitch information, input the second feature map to a second extended neural network block that restores the feature map to restore the first feature map, and features based on the pitch information An extension factor of a first extended neural network block for reconstructing an audio signal from a map may be determined, and an audio signal may be restored from the first feature map using the first extended neural network block.

The processor may up-sample the first feature map to extend a dimension, and input the up-sampled first feature map to the first scalable neural network block to determine the audio signal.

The expansion factor of the first extended neural network block may be determined by approximating an accommodation area of the first extended neural network block with the pitch information.

The expansion factor of the second extended neural network block may be fixed to a predetermined value, and the accommodation area of the second extended neural network block may be determined by the expansion factor of the second extended neural network block.

According to an embodiment of the present invention, by variably determining the expansion factor of the extended neural network model using pitch information of the audio signal, it is possible to effectively remove the long-term redundancy inherent in the audio signal in the neural network-based audio encoding and decoding process. .

In addition, according to an embodiment of the present invention, by determining the expansion factor of the extended neural network model using pitch information of the audio signal, the quality of the audio signal restored through the variable neural network encoding and decoding model according to the characteristics of the audio signal is improved. It can be improved, and computational complexity can be reduced compared to the conventional scalable neural network model having a fixed expansion factor.

1 is an encoder and a decoder according to an embodiment of the present invention.
2 is a diagram illustrating a process of processing an encoding method and a decoding method according to an embodiment of the present invention.
3A and 3B are diagrams illustrating a hierarchical structure of a neural network model according to an embodiment of the present invention.
4 is a diagram illustrating a hierarchical structure of a neural network model determined according to pitch information according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is an encoder and a decoder according to an embodiment of the present invention.

The present invention determines a receptive field of a neural network model based on artificial intelligence using pitch information of an audio signal in encoding and decoding an audio signal, and encoding and decoding an audio signal through the neural network model By doing so, the present invention may relate to a technique for reducing short-term redundancy and long-term redundancy occurring during encoding and decoding of audio signals.

The encoder and decoder for performing the encoding method and the decoding method of the present invention may include a processor such as a smart phone, a desktop computer, or a laptop computer. The encoder and the decoder may be different electronic devices or may be the same electronic device.

The encoding and decoding model of the present invention may be a neural network model based on deep learning. For example, the encoding and decoding model may be an autoencoder configured with a convolutional neural network. The encoding and decoding models that can be used in the present invention are not limited to the described examples, and various types of neural network models can be used.

The neural network model may include an input layer, a hidden layer, and an output layer, and each layer may include a plurality of nodes. A node of each layer may be calculated as a product of the nodes of the previous layer and a matrix composed of an arbitrary weight. The weight of the matrix between each layer may be updated during training of the neural network model. In particular, in the case of a convolutional neural network, a filter, which is a weight matrix, is used to calculate a feature map for each layer. In general, a feature map of each layer is calculated through a plurality of filters, and the number of filters used is called the number of channels.

The neural network model generates output data for input data, the input layer may correspond to input data of the neural network model, and the output layer may correspond to output data of the neural network model. In the present invention, the input data and the output data may be vectors representing an audio signal having a constant length (frame), and when configured with a plurality of audio frames, the input and output data may be expressed as a two-dimensional matrix.

The feature map for each layer of the neural network model may be a one-dimensional vector, a two-dimensional matrix, or a multi-dimensional tensor indicating characteristics of an audio signal. For example, the feature map may be input data or data obtained as a result of an operation between a feature map of a previous layer and a weight filter of each layer. The reception area of the neural network model is the number of input nodes used to calculate the value of each node of the output layer, and is determined by the length of the weight filter and the number of layers constituting the learning model. The receptive area of the scalable neural network model may be further determined by an expansion factor. The reception area of the neural network model according to the expansion factor will be described later with reference to FIGS. 3A, 3B, and 4 .

The number of channels of the input signal depends on the signal representation of the original signal. For example, the number of channels is 1 and 2 for a mono and stereo signal of an audio signal, respectively, and the number of channels corresponds to 3 for an RGB color image signal. . Meanwhile, in the convolutional neural network, the number of channels in the output feature map is determined by the number of convolution filters used to calculate the output feature map.

The pitch information of the audio signal may refer to information indicating periodicity of the audio signal. As an example, pitch information is used to model long-term redundancy of a signal in a general audio compressor to express periodicity inherent in an input audio signal, and may mean a pitch lag for each frame. That is, the pitch information may be defined as a difference between the previous time and the specific time, which are generally found through a method of searching for a viewpoint having the greatest correlation between the audio signal of a specific viewpoint and the audio signal of the previous viewpoint. In this case, the search time may include time points inside the corresponding audio signal frame and time points of previous frames.

Referring to FIG. 1 , the encoder of the present invention may generate a bitstream by encoding an input signal, and the decoder may generate an output signal from the bitstream received from the encoder. The input signal may mean an original audio signal received by the encoder, and the output signal may mean an audio signal reconstructed from the decoder. A detailed operation of encoding and decoding an audio signal using the learning model will be described later with reference to FIG. 2 .

2 is a diagram illustrating a process of processing an encoding method and a decoding method according to an embodiment of the present invention.

According to an embodiment of the present invention, including a channel transformation block 201, a first scalable neural network block 202, a downsampling block 203, a second scalable neural network block 204, a channel transformation block 205 A neural network model may be used for encoding the input signal.

In the pitch information extraction 206 operation, the encoder 101 may extract pitch information of the audio signal. For example, the encoder 101 extracts pitch information by calculating a normalized autocorrelation for an audio signal frame for each viewpoint corresponding to a predetermined pitch delay search range, and then searching for a viewpoint having a maximum value. can do. A detailed method of extracting pitch information may not be limited to the described example.

In the quantization 207 operation, the encoder 101 may quantize the extracted pitch information to a value expressible by a predetermined number of bits. Also, the encoder 101 may convert the quantized pitch information into a bitstream.

The encoder 101 may determine an extension factor of the first extended neural network block 202 extension block based on the quantized pitch information. The reception area of the first extended neural network block 202 may be determined by the length of the filter, the number of layers, and the expansion factor. The length and the number of layers of the filter are predetermined in the neural network model design stage, while the expansion factor is applied every audio frame. is calculated according to the quantized pitch information.

The first extended neural network block 202 is a convolutional neural network that calculates a new output feature map from an input feature map, and may be a neural network block having an extension factor variably determined according to pitch information. The first extended neural network block 202 may be distinguished from the second extended neural network block 204 in which an extension factor is fixed.

In the present invention, the first extended neural network block 202 based on pitch information without excessively increasing the number of layers of the neural network block and the length of the filter in order to widen the receiving area, unlike the conventional extended neural network having a fixed extension factor. By variably determining the expansion factor of , it is possible to obtain a sufficient accommodating area required for long-term modeling with a relatively small number of layers, thereby reducing computational complexity.

As an example, the channel transformation block 201 and the channel transformation block 205 used in the encoder 101, the downsampling neural network block, and the first extended neural network block 202 and the second extended neural network block 204 are convolutional neural networks. It may be a component of the encoder 101 of the auto-encoder using and the second scalable neural network block 204 may be a component of the encoder 101 of an auto-encoder using a convolutional neural network.

As an example, in the encoder 101, the channel transformation block 201 applies a convolution with a plurality of filters (corresponding to the number of channels in the output feature map) to an input audio signal that has only a single or two channels to be included in the input signal. It may be a neural network block that extracts various features and outputs a channel-transformed feature map.

The first extended neural network block 202 used in the encoder 101 applies extended convolution having an extension factor based on the quantized pitch information to the channel transformed feature map output by the channel transformation block 201 to the audio signal. It may be a neural network block that outputs the first feature map from which the inherent long-term redundancy is removed. The first feature map is a feature map output from the first extended neural network block used in the encoder 101, and can be used as input data of the second extended neural network block, and is output data of the second extended neural network block. 2 It can be distinguished from the feature map. The second feature map may mean a feature map in which the first feature map is processed by the second extended neural network block.

The downsampling block 203 used in the encoder 101 applies strided convolution or convolution combined with pooling, etc. to the first feature map output by the first extended neural network block 202 . Thus, it may be a neural network block that outputs a down-sampled feature map in which the dimension of the input feature map is reduced.

The second extended neural network block 204 used in the encoder 101 removes short-term redundancy inherent in the audio signal by applying extended convolution with a fixed extension factor to the feature map output by the down-sampling neural network block. It may be a neural network block that outputs a map. The encoder 101 may determine the second feature map from the down-sampled first feature map using the second extended neural network block. The second feature map may have a smaller size than the first feature map.

The channel transform block 205 used in the encoder 101 is a channel transformed latent for quantization by applying convolution using a predetermined number of filters to the second feature map output by the second extended neural network block 204 . ) may be a neural network block that outputs a feature map.

The channel conversion block 205 may convert a channel of the second feature map. That is, the channel of the second feature map is set to correspond to the filter length of the second extended neural network block (eg, the number of weight filters used to determine the weight filter of the l+1th layer in the lth layer). Therefore, the channel conversion block 205 may convert the channel of the second feature map into the channel of the input signal.

In the quantization 208 operation, the encoder 101 may quantize the latent feature map output from the channel transform block 205 into a value expressible by a predetermined number of bits. In addition, the quantized latent feature map can be converted into a bitstream.

In the multiplexing 209 operation, the encoder 101 multiplexes the quantized pitch information bitstream and the quantized latent feature map bitstream to output the entire bitstream.

According to an embodiment of the present invention, including a channel transformation block 212, a first scalable neural network block 215, an up-sampling block 214, a second scalable neural network block 213, and a channel transformation block 216 A neural network model may be used for decoding an audio signal.

In the de-multiplexing 210 operation, the decoder 102 de-multiplexes the entire bitstream received from the encoder 101 to extract a quantized pitch information bitstream and a quantized latent feature map bitstream, respectively.

In the de-quantization 217 operation, the decoder 102 de-quantizes the quantized pitch information bitstream to extract the quantized pitch information. In the de-quantization 211 operation, the decoder 102 de-quantizes the quantized latent feature map bitstream to extract the quantized latent feature map.

The channel transform block 212 used in the decoder 102 restores the short-term redundancy inherent in the audio signal by applying convolution using a predetermined number of filters to the latent feature map quantized through the inverse quantization process. 2 It may be a neural network block that outputs a feature map.

The channel conversion block 212 may convert a channel of the second feature map. Specifically, in the channel transformation block 212, the channel of the second feature map is the length of the filter of the second extended neural network block (eg, in the l-th layer, the weight used to determine the weight filter of the l+1th layer) The channel of the second feature map may be transformed to correspond to the number of filters).

The second extended neural network block 213 used in the decoder 102 restores the down-sampled feature map by applying the extended convolution having a fixed extension factor to the second feature map output by the channel transformation block 212 . It may be a neural network block that

The up-sampling block 214 used in the decoder 102 performs deconvolution or sub-pixel convolution on the down-sampled feature map output by the second extended neural network block 213 . It may be a neural network block that restores the first feature map that extends the dimension of the input feature map by applying it.

The first extended neural network block 215 used in the decoder 102 applies extended convolution having an extension factor based on the quantized pitch information to the first feature map output from the up-sampling block 214 to the audio signal. It may be a neural network block that outputs a channel-transformed feature map in which the inherent long-term redundancy is restored.

The channel transformation block 216 used in the decoder 102 applies a convolution having the same number of filters as the number of channels of the original input audio signal to the channel-transformed feature map output by the first extended neural network block to input audio signal. It may be a neural network block that restores

The channel conversion block 216 may convert the channel of the reconstructed output signal. As an example, since the channel of the reconstructed output signal may correspond to the filter length of the first extended neural network block (the number of weight filters used to determine the weight filter of the l+1th layer in the lth layer), the input signal To correspond to the channel of , the channel conversion block 216 may convert the channel of the output signal into a mono or stereo channel.

The model parameters such as the sum filter and bias of all neural network blocks used in the encoder 101 and the decoder 102 are the audio signal restored by the decoder 102 and the input to the encoder 101 . It can be trained by comparing the raw audio signals. That is, the channel transformation blocks 201 and 205 used in the encoder 101 and the decoder 102 so that the difference between the audio signal restored by the decoder 102 and the audio signal input to the encoder 101 is minimized. , 212 and 216 , the down-sampling neural network 203 , the up-sampling neural network 214 , and the model parameters of the first extended neural network blocks 202 and 215 and the second extended neural network blocks 204 and 213 may be updated.

For example, according to the expansion factor, the accommodation areas of the first extended neural network blocks 202 and 215 and the second extended neural network blocks 204 and 213 may be determined according to Equation 1 below.

In Equation 1, r means the accommodation area of the extended neural network block (202, 204, 215, 213), and L is the number of all layers included in the extended neural network block (202, 204, 215, 213). can k _l may mean the length of the convolution filter between the (l+1)-th layers in the l-th layer. k _l may have the same value regardless of the layer. d _l may mean an extension factor of the l-th layer. As an example, d _l may be determined according to Equation 2 below. For example, when the number of layers and the length of the weight filter are fixed, the accommodation area of the extended neural network block may be expressed as a function of the expansion factor as shown in Equation (1).

Referring to Equation 2, the extension factor of the l-th layer may be determined to be twice the extension factor of the (l-1)-th layer. However, the relationship between the extension factor of the (l-1)-th layer and the extension factor of the l-th layer may not be limited to the described example.

As an example, the expansion factor for each layer of the first extended neural network block may be determined based on pitch information of the audio signal, and the expansion factor for each layer of the second extended neural network block 204 and 213 is the audio signal and Regardless, it can be determined as a preset value.

For example, in the expansion factor determination process (204, 217) of the encoder 101 and the decoder 102, in order to determine the expansion factor of the first extended neural network block (202, 215) based on the pitch information of the audio signal , Equations 3 and 4 below may be used.

p는 오디오 신호의 양자화된 피치 지연을 의미할 수 있다. 본 발명에서, 장기간 중복성을 줄이기 위하여, 제1 확장형 신경망 블록(202, 215)의 수용 영역은 오디오 신호의 피치 지연과 일치하도록 결정될 수 있다. In Equation 3, r means the receiving area of the first extended neural network block (202, 204, 215, 213),

p may mean a quantized pitch delay of an audio signal. In the present invention, in order to reduce long-term redundancy, the receiving area of the first extended neural network blocks 202 and 215 may be determined to match the pitch delay of the audio signal.

은 반올림 연산(rounding operation)을 의미할 수 있다. 첫번째 계층의 확장 인자(d₁)로부터 수학식 2에 정의된 관계를 통해서 나머지 계층의 확장 인자를 구할 수 있다. In Equation 4, d ₁ may mean an extension factor of the first layer of the first extended neural network blocks 202 and 215 . k may mean the length of the convolution filter between the l+1th layer in the lth layer. L may mean the number of all layers included in the first extended neural network blocks 202 and 215 .

may mean a rounding operation. From the expansion factor (d ₁ ) of the first layer, the expansion factor of the remaining layers can be obtained through the relationship defined in Equation (2).

In the channel conversion process 219 , the decoder 102 may convert the channel of the reconstructed output signal. As an example, since the channel of the reconstructed output signal may correspond to the filter length of the first extended neural network block (the number of weight filters used to determine the weight filter of the l+1th layer in the lth layer), the input signal To correspond to the channel of , the decoder 102 may convert the channel of the output signal into a mono or stereo channel.

3A and 3B are diagrams illustrating a hierarchical structure of a learning model according to an embodiment of the present invention.

In each of FIGS. 3A and 3B , the filter length of all layers 301-303, 311-313 is, in the l-th layer, the weight filter used to determine the weight filter 304, 314 of the l+1-th layer. (the number of 304, 314) can be determined to be three. 3A shows a process in which the weight filter 304 of the output layer is determined when the receiving area 305 of the learning model is 5 and the expansion factor of the learning model is determined to be 1 in all layers 301-303 hierarchical structure It is a drawing shown as

Referring to FIG. 3A , three weight filters 304 in the input layer 301 may be used to determine the weight filter 304 of the hidden layer 302 , and in the hidden layer 302 , three weight filters 304 . A filter 304 may be used to determine the weight filter 304 of the output layer 303 . Referring to FIG. 3A , five weight filters 304 in the input layer 301 may be used to determine one weight filter 304 in the output layer 303 . That is, in FIG. 3A , the reception area 305 of the learning model may be determined to be 5 .

3B shows a process in which the weight filter 314 of the output layer is determined when the accommodation area 315 of the learning model is 5, the expansion factor of the learning model is 1 in the hidden layer, and 2 in the output layer. It is a drawing shown as That is, it may be a case in which the extension factor increases according to the layer. As an example, FIG. 3B may be an example of an extended convolutional neural network, and FIG. 3A may be an example of a general convolutional neural network.

Referring to FIG. 3B , three weight filters 314 in the input layer 311 may be used to determine the weight filters 314 of the hidden layer 312 , and in the hidden layer 312 , three weight filters 314 . A filter 314 may be used to determine the weight filter 314 of the output layer 313 . Referring to FIG. 3B , seven weight filters 314 in the input layer 311 may be used to determine one weight filter 314 in the output layer 313 . That is, the accommodation area 315 of the learning model of FIG. 3B may be determined as 7 .

4 is a diagram illustrating a hierarchical structure of a learning model determined according to pitch information according to an embodiment of the present invention.

p)은 3으로 결정되고, 모든 계층(401-403)의 필터 길이가 2로 결정된 경우일 수 있다. 도 4를 참조하면, 피치 지연에 기초하여, 첫번째 계층(401)의 확장 인자는 1로 결정될 수 있다. 그리고, 입력 계층(401)의 확장 인자에 따라 은닉 계층(402)의 확장 인자는 2로 결정되고, 출력 계층의 확장 인자는 4로 결정될 수 있다. 이에 따라, 학습 모델의 수용 영역은 4로 결정될 수 있다. 4, pitch delay 405 (eg:

p) is determined to be 3, and the filter length of all layers 401 to 403 may be determined to be 2. Referring to FIG. 4 , the expansion factor of the first layer 401 may be determined to be 1 based on the pitch delay. And, according to the expansion factor of the input layer 401, the expansion factor of the hidden layer 402 may be determined to be 2, and the expansion factor of the output layer may be determined to be 4. Accordingly, the reception area of the learning model may be determined to be 4.

Meanwhile, the method according to the present invention is written as a program that can be executed on a computer and can be implemented in various recording media such as magnetic storage media, optical reading media, and digital storage media.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or combinations thereof. Implementations may be implemented for processing by, or for controlling the operation of, a data processing device, eg, a programmable processor, computer, or number of computers, a computer program product, ie an information carrier, eg, a machine readable storage It may be embodied as a computer program tangibly embodied in an apparatus (computer readable medium) or a radio signal. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be written as a standalone program or in a module, component, subroutine, or computing environment. may be deployed in any form, including as other units suitable for use in A computer program may be deployed to be processed on one computer or multiple computers at one site or to be distributed across multiple sites and interconnected by a communications network.

Processors suitable for processing a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from either read-only memory or random access memory or both. Elements of a computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. In general, a computer may include one or more mass storage devices for storing data, for example magnetic, magneto-optical disks, or optical disks, receiving data from, sending data to, or both. may be combined to become Information carriers suitable for embodying computer program instructions and data are, for example, semiconductor memory devices, for example, magnetic media such as hard disks, floppy disks and magnetic tapes, Compact Disk Read Only Memory (CD-ROM). ), an optical recording medium such as a DVD (Digital Video Disk), a magneto-optical medium such as a floppy disk, a ROM (Read Only Memory), a RAM , Random Access Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and the like. Processors and memories may be supplemented by, or included in, special purpose logic circuitry.

In addition, the computer-readable medium may be any available medium that can be accessed by a computer, and may include both computer storage media and transmission media.

While this specification contains numerous specific implementation details, these are not to be construed as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. should be understood Certain features that are described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although features operate in a particular combination and may be initially depicted as claimed as such, one or more features from a claimed combination may in some cases be excluded from the combination, the claimed combination being a sub-combination. or a variant of a sub-combination.

Likewise, although acts are depicted in the figures in a particular order, it should not be construed that all acts shown must be performed or that such acts must be performed in the specific order or sequential order shown to achieve desirable results. In certain cases, multitasking and parallel processing may be advantageous. Further, the separation of the various device components of the above-described embodiments should not be construed as requiring such separation in all embodiments, and the program components and devices described may generally be integrated together into a single software product or packaged into multiple software products. You have to understand that you can.

On the other hand, the embodiments of the present invention disclosed in the present specification and drawings are merely presented as specific examples to aid understanding, and are not intended to limit the scope of the present invention. It will be apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

101: encoder
102: decoder

Claims (16)

A method of encoding an audio signal using a learning model, the method comprising:
extracting pitch information for the audio signal;
determining an extension factor for a receptive field of a first scalable neural network block for extracting a feature map from the audio signal based on the pitch information;
generating a first feature map of the audio signal using a first extended neural network block in which the expansion factor is determined;
determining a second feature map by inputting the first feature map to a second extended neural network block that processes the first feature map;
converting the second feature map and the pitch information into a bitstream
A coding method comprising a. According to claim 1,
The step of generating the first feature map comprises:
generating the first feature map by converting the channel of the audio signal and inputting it to the first extended neural network block;
The step of determining the second feature map comprises:
The encoding method further comprising the step of transforming the determined number of channels of the second feature map. According to claim 1,
The step of determining the second feature map,
Down-sampling for reducing the dimension of the first feature map is performed on the first feature map, and the down-sampled first feature map is input to the second scalable neural network block to provide the second feature A coding method for determining a map. The method of claim 1,
The step of determining the expansion factor comprises:
and determining the expansion factor by approximating an accommodation area of the first extended neural network block with the pitch information. According to claim 1,
The expansion factor of the second scalable neural network block is predetermined as a fixed value,
The receiving area of the second extended neural network block is,
The encoding method is determined according to an extension factor of the second scalable neural network block. According to claim 1,
Further comprising the step of quantizing each of the second feature map and the pitch information,
The step of converting to the bitstream comprises:
An encoding method of multiplexing the quantized second feature map and pitch information into a bitstream. A method of decoding an audio signal using a learning model, the method comprising:
extracting a second feature map of the audio signal and pitch information of the audio signal from the bitstream received from the encoder;
restoring a first feature map by inputting the second feature map to a second extended neural network block for restoring a feature map;
determining an extension factor for a receptive field of a first scalable neural network block for reconstructing an audio signal from a feature map based on the pitch information;
reconstructing an audio signal from the first feature map using a first extended neural network block in which the extension factor is determined;
A decryption method comprising 8. The method of claim 7,
Restoring the first feature map comprises:
The first feature map is restored by transforming the number of channels in the second feature map and inputting it to the second extended neural network block,
The step of restoring the audio signal comprises:
The decoding method further comprising the step of converting the number of channels of the reconstructed audio signal to be the same as the number of channels of the input signal of the encoder. 8. The method of claim 7,
Restoring the audio signal comprises:
Up-sampling is performed on the first feature map to extend a dimension of the first feature map, and the up-sampled first feature map is input to the first extended neural network block to generate the audio signal. Determining the decryption method. 8. The method of claim 7,
The expansion factor is
In the encoder, the decoding method is determined by approximating an accommodation region of the first extended neural network block with the pitch information. 8. The method of claim 7,
The expansion factor of the second scalable neural network block is predetermined as a fixed value,
The receiving area of the second extended neural network block is,
The decoding method is determined according to an expansion factor of the second extended neural network block. 8. The method of claim 7,
The step of extracting the second feature map and the pitch information of the audio signal,
The decoding method further comprising the step of inverse-quantizing the second feature map and the pitch information, respectively. An encoder for performing an audio signal encoding method, comprising:
The encoder includes a processor,
The processor is
extracting pitch information for the audio signal, and determining an extension factor for a receptive field of a first extended neural network block for extracting a feature map from the audio signal based on the pitch information; A first feature map of the audio signal is generated using the first extended neural network block in which the extension factor is determined, and the first feature map is input to a second extended neural network block that processes the first feature map. to determine a second feature map, and convert the second feature map and the pitch information into a bitstream,
Encoder. 14. The method of claim 13,
The processor is
Down-sampling for reducing the dimension of the first feature map is performed on the first feature map, and the down-sampled first feature map is input to the second scalable neural network block to provide the second feature The encoder that determines the map. 14. The method of claim 13,
The processor is
and determining the expansion factor by approximating an accommodation area of the first extended neural network block with the pitch information. 14. The method of claim 13,
The expansion factor of the second scalable neural network block is predetermined as a fixed value,
The receiving area of the second extended neural network block is,
The encoder, which is determined according to an extension factor of the second extended neural network block.

Images