KR20240062924A - Method for encoding/decoding audio signal and apparatus for performing the same - Google Patents

Abstract

An audio signal encoding/decoding method and a device for performing the same are disclosed. An audio signal encoding method according to an embodiment includes obtaining a full-band input signal; The full-band input signal is converted into a first feature vector corresponding to a first sub-band signal using an encoder neural network including a plurality of encoding layers. and extracting a second feature vector corresponding to the second subband signal; And compressing the first feature vector and the second feature vector to generate a first code vector corresponding to the first feature vector and a second code vector corresponding to the second feature vector. movement; and generating a bitstream by quantizing the first code vector and the second code vector.

Description

Audio signal encoding/decoding method and device for performing the same {METHOD FOR ENCODING/DECODING AUDIO SIGNAL AND APPARATUS FOR PERFORMING THE SAME}

The disclosure below relates to an audio signal encoding/decoding method and a device for performing the same.

Spectral sub-bands may not have the same perceptual relevance. In audio coding, sub-band signals of the original full-band signal are used to efficiently perform bitrate assignment and signal reconstruction. A coding method to independently control signals may be required.

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

One embodiment improves coding quality by independently controlling reconstruction and bitrate allocation for the core band signal and high band signal of the original full-band signal. You can do it.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An audio signal encoding method according to an embodiment includes obtaining a full-band input signal; The full-band input signal is converted into a first feature vector corresponding to a first sub-band signal using an encoder neural network including a plurality of encoding layers. and extracting a second feature vector corresponding to the second subband signal; An operation of compressing the first feature vector and the second feature vector to generate a first code vector corresponding to the first feature vector and a second code vector corresponding to the second feature vector. ; and generating a bitstream by quantizing the first code vector and the second code vector.

The first sub-band signal includes a high band signal of the full-band input signal, and the second sub-band signal includes a downsampled core band signal of the full-band input signal. It can be included.

Each of the plurality of encoding layers may encode the output of the previous encoding layer.

The extracting operation includes extracting an output of an intermediate encoding layer among the plurality of encoding layers as the first feature vector; and extracting the output of the last encoding layer among the plurality of encoding layers as the second feature vector.

Each of the plurality of encoding layers may include a 1-dimensional convolution layer.

Among the plurality of encoding layers, except for one or more encoding layers located behind the middle encoding layer, the remaining encoding layers have the same stride value, and the one or more encoding layers have a stride value greater than the remaining encoding layers. You can have

The stride value of the remaining encoding layers may be 1.

An audio signal decoding method according to an embodiment includes a reconstructed first feature vector corresponding to a first subband signal of an original full-band signal and the original full-band signal. Obtaining a reconstructed second feature vector corresponding to a second subband signal of the signal; Restoring the second sub-band signal based on the restored second feature vector; Restoring the first band signal based on the restored first feature vector and the restored second feature vector; and restoring the original full-band signal based on the restored first sub-band signal and the restored second sub-band signal.

The first sub-band signal may include a high-band signal of the original full-band signal, and the second sub-band signal may include a downsampled core band signal of the original full-band signal.

The operation of restoring the second subband signal includes encoding the restored second feature vector using a decoder neural network including a plurality of decoding layers, and Each of the decoding radars can encode the output of the previous decoding layer.

The operation of restoring the first subband signal includes encoding the restored first feature vector and the restored second feature vector using a decoder neural network including a plurality of decoding layers, and the plurality of decoding layers Each of the layers encodes the output of the previous decoding layer, and an intermediate decoding layer among the plurality of decoding layers may encode the output of the previous decoding layer using the restored second feature vector.

An apparatus for encoding an audio signal according to an embodiment includes a memory including instructions; and a processor electrically connected to the memory and configured to execute the instructions. When the instructions are executed by the processor, the processor may perform a plurality of operations. The plurality of operations may include obtaining a full-band input signal; The full-band input signal is converted into a first feature vector corresponding to a first sub-band signal using an encoder neural network including a plurality of encoding layers. and extracting a second feature vector corresponding to the second subband signal; An operation of compressing the first feature vector and the second feature vector to generate a first code vector corresponding to the first feature vector and a second code vector corresponding to the second feature vector. ; and generating a bitstream by quantizing the first code vector and the second code vector.

The first sub-band signal includes a high band signal of the full-band input signal, and the second sub-band signal includes a downsampled core band signal of the full-band input signal. It can be included.

Each of the plurality of encoding layers may encode the output of the previous encoding layer.

The extracting operation includes extracting an output of an intermediate encoding layer among the plurality of encoding layers as the first feature vector; and extracting the output of the last encoding layer among the plurality of encoding layers as the second feature vector.

Each of the plurality of encoding layers may include a 1-dimensional convolution layer.

Among the plurality of encoding layers, except for one or more encoding layers located behind the middle encoding layer, the remaining encoding layers have the same stride value, and the one or more encoding layers have a stride value greater than the remaining encoding layers. You can have

The stride value of the remaining encoding layers may be 1.

Figure 1 is a diagram for explaining an encoder and decoder according to an embodiment.
Figure 2 is a diagram for explaining an autoencoder according to an embodiment.
Figure 3 is a diagram for explaining an encoder neural network and a decoder neural network according to an embodiment.
Figure 4 is a flowchart for explaining the operation of an encoder according to an embodiment.
Figure 5 is a flowchart for explaining the operation of a decoder according to an embodiment.
Figure 6 is a schematic block diagram of an encoder according to one embodiment.
Figure 7 is a schematic block diagram of a decoder according to one embodiment.
Figure 8 is a schematic block diagram of an electronic device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. In this specification, terms such as "comprise" or "have" are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, but are not intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

The term '~unit' used in this document refers to software or hardware components such as FPGA or ASIC, and '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. For example, '~part' refers to software components, object-oriented software components, components such as class components and task components, processes, functions, properties, procedures, May include subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card. Additionally, '~ part' may include one or more processors.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

Figure 1 is a diagram for explaining an encoder and decoder according to an embodiment.

Referring to FIG. 1, according to one embodiment, the coding system 100 may include an encoder 110 and a decoder 160. The encoder 110 may encode an input audio signal using an encoder neural network and output a bitstream. The input signal may be a full-band time domain signal. The encoder 110 will be described in detail with reference to FIGS. 2 to 4.

The decoder 160 may receive a bitstream from the encoder 110, decode the encoded signal using a decoder neural network, and output a reconstructed signal corresponding to the input signal. The decoder 160 will be described in detail with reference to FIGS. 2, 3, and 5.

Figure 2 is a diagram for explaining an autoencoder according to an embodiment.

Referring to FIG. 2, according to one embodiment, the autoencoder 200 may include an encoder 210 (eg, the encoder and decoder 260 of FIG. 1).

The encoder 210 may include a plurality of encoding layers 212. The encoder 210 can encode the input signal by learning hidden features from the input signal using a plurality of encoding layers 212 (e.g., one-dimensional convolution layers) as shown in Equation 1. there is.

[Equation 1]

In equation 1, represents the input signal, represents the number of plurality of encoding layers 212, is a feature vector, May represent a plurality of encoding layers or an encoding function of a plurality of encoding layers.

The encoder 210 may obtain a quantized feature vector (eg, bitstring) by applying a quantization function to the feature vector. The quantization function assigns the floating-point values of the feature vector to a finite set of quantization bins (e.g., 32 centroids for a 5-bit system), thereby creating a quantized feature vector. can be obtained. The encoder 210 may compress the quantized feature vector through entropy coding such as Huffman coding.

The decoder 260 may include a plurality of decoding layers 262. The decoder 260 restores the input signal by dequantizing the quantized feature vector as shown in Equation 2 and decoding the recovered feature vector step by step using a plurality of decoding layers 262. can do.

[Equation 2]

In equation 2 represents a quantized feature vector (e.g. a bitstring), represents the input signal (e.g. the original signal), represents the reconstructed input signal, May represent a plurality of decoding layers or a decoding function of a plurality of decoding layers. Decoding layers may correspond to encoding layers. For example, the last decoding layer of the decoder 260 may correspond to the first encoding layer of the encoder 210.

Figure 3 is a diagram for explaining an encoder neural network and a decoder neural network according to an embodiment.

Referring to FIG. 3, according to one embodiment, a neural network based coding system 300 includes an encoder 310 (e.g., encoder 110 of FIG. 1) and a decoder 360 (e.g., It may include the decoder 160 of FIG. 1).

The encoder 310 uses an encoder neural network 311 including a plurality of encoding layers (e.g., one-dimensional convolution layers) to encode an input signal ( ) can be encoded. Input signal ( ) may be a full-band input time-domain signal. Input signal ( ) can be defined as the sum of two subband signals as shown in Equation 3.

[Equation 3]

In Equation 3, the first subband signal ( ) is the input signal ( ), and represents a high-pass filtered version of the second subband signal ( ) is the input signal ( ) can represent a down-sampled version. The second subband signal is subjected to interpolation-based upsampling (in the inference phase). ) can be used to recover the original temporal resolution. The reconstruction loss for neural network training is a down-sampled version ( ) can be calculated using .

The encoder neural network 311 may be composed of M (e.g., M is a natural number) two stages 31 and 36 (e.g., cascaded stages).

The first stage 31 may include M (e.g., M is a natural number) encoding layers (e.g., one-dimensional convolution layers). The first encoding layer of the first stage 31 is the input signal ( ), and each of the encoding layers of the first stage 31 can encode the output of the previous encoding layer step by step. During the encoding process of the first stage 31, the temporal dimension may remain the same due to the stride value (e.g. 1) and zero padding of the encoding layers of the first stage 31. . Unlike the time dimension, the channel dimension can increase by the number of filters. The last encoding layer 311_1 of the first stage 31 (e.g., the intermediate encoding layer of the encoder neural network 311) is the first subband signal ( ), the first feature vector corresponding to ( ) can be output. Encoding layers of the first stage 31 may have the same stride value (eg, 1).

The second stage 36 may include N (e.g., N is a natural number) encoding layers (e.g., one-dimensional convolution layers). The first encoding layer of the second stage 36 is the first feature vector ( ) can be received as input. Each encoding layer of the second stage 36 can encode the output of the previous encoding layer step by step. Stride values of the encoding layers of the second stage 36 may be different. For example, some may have a first stride value (e.g., 1), and others may have a second stride value (e.g., a natural number greater than 1). Due to the stride values of the encoding layers of the first stage 36, the decimating factor (or down-sampling ratio) may be expressed as Equation 4 below.

[Equation 4]

In equation 4: is the decimating factor of the second stage 36 ( ), is the participating layer It can represent the stride of (e.g. encoding layer).

The last encoding layer of the second stage 36 is the second subband signal ( ) and the second feature vector corresponding to ( ) can be output. The second feature vector ( ) is the decimating factor of the second stage 36 ( ) Based on temporal decimation ( ) may lose the high frequency contents corresponding to it. However, the second sub-band signal ( ) may not be affected.

The encoder neural network 311 can compress and transmit information to the decoder 360 through a skip connection (autoencoder based skip connection) using an autoencoder (e.g., skip autoencoders 331, 336). . The first skip autoencoder 331 and the second skip autoencoder 336 each have an encoder ( ), decoder( ), and quantization module ( ) may include.

The first skip autoencoder 331 uses the first feature vector ( ) as input, and the second skip autoencoder 336 receives a second feature vector ( ) can be received as input. Encoders of skip autoencoders (331, 336) ( ) Each is an input feature vector, e.g. a first feature vector ( ), second feature vector ( )) can be encoded. Encoders ( ) performs compression on the input feature vector, but may not perform downsampling on the input feature vector. For example, encoders ( ) can reduce the channel dimension to 1, as shown in Equation 5 and Equation 6.

[Equation 5]

[Equation 6]

In Equation 5 and Equation 6, is the first feature vector ( ) represents the first code vector corresponding to is the second feature vector ( ) may indicate a second code vector corresponding to .

Skip autoencoders 331 and 336 can reduce the data rate while maintaining temporal resolution through channel reduction.

Quantization module of the first skip autoencoder 331 ( ) is the first code vector ( ), and the quantization module of the second skip autoencoder 336 generates a second code vector ( ) can be performed. Quantization module ( ) is a feature vector using a scalar feature assignment matrix (learned controids) as a feature vector (e.g., the first feature vector ( ), second feature vector ( ) can be assigned to each of the feature values. In the inference stage, the quantization module ( ) is a scalar feature assignment matrix ( ) to make a non-differentiable "hard" assignment (e.g. ) can be performed. Scalar feature assignment matrix ( ) may be a one-hot vector that selects the closest centroid for the ith feature. To circumvent non-differentiable processes, in the training phase, a soft version of the allocation matrix ( ) can be used. Backpropagation errors may occur due to differences between the inference phase and the training phase. The discrepancy between the soft allocation result and the hard allocation result is determined by the temperature factor ( ) can be handled by annealing.

[Equation 7]

Given a feature vector and centroids, a distance matrix representing the absolute difference between each element of the feature vector and each centroid (e.g. ) can be created. From the distance matrix, a probabilistic vector for the ith element of the feature vector can be derived as shown in Equation 8.

[Equation 8]

First subband signal ( ) and the second subband signal ( ), the number of learned centroids, and/or bitrates can be learned and controlled individually.

Decoder of the first skip autoencoder 331 ( ) is the quantized first code vector ( ), by decoding the first feature vector ( ), and the decoder of the second skip autoencoder 336 ( ) is the quantized second code vector ( ), by decoding the second feature vector ( ) can be restored.

The decoder 360 uses the first decoder neural network 361 to generate a first subband signal ( ) is restored, and the second subband signal ( ) can be restored. Each of the first decoder neural network 361 and the second decoder neural network 366 may include a plurality of decoding layers. Each of the plurality of decoding layers may decode the output of the previous decoding layer.

The first decoder neural network 361 may use nearest-neighbors-based up-sampling to compensate for the loss of temporal resolution. That is, the first decoder neural network 361 may perform a band extention (BE) process. The first decoder neural network 361 can increase temporal resolution through upsampling, as shown in Equation 9.

[Equation 9]

In equation 9, is the participation layer It may indicate the scaling factor (e.g., decoding layer).

The middle decoding layer 361_1 of the first decoder neural network 361 is the restored first feature vector output from the first skip autoencoder 331 ( ), you can decode the output of the previous decoding layer. For example, the restored first feature vector ( ) is the output of the previous decoding layer along the channel dimension ( ) and is concatenated with a concatenated vector ( ) can be input to the intermediate decoding layer (361_1). The middle decoding layer (361_1) uses the input vector ( ) can be reduced in channel dimension. For example, the middle decoding layer 361_1 may reduce the channel dimension from 2C (e.g., C is a natural number) to C. The decoding process of the middle decoding layer 361_1 can be expressed as Equation 10 below.

[Equation 10]

The decoding layers of the second decoder neural network 366 share the same depth as the corresponding encoding layers (e.g., the encoding layers of the encoder neural network 311), thereby generating a second subband signal ( ) can be restored.

Encoder neural network 311 and/or decoder neural networks 361, 366 generate quantized code vectors ( , It can be learned through a loss function (e.g., an objective function) to reach the target entropy of ).

Network loss (e.g., loss function) may be composed of reconstruction loss (or reconstruction error) and entropy loss, as shown in Equation 11.

[Equation 11]

In equation 10, represents the reconstruction loss, can represent entropy loss.

Reconstruction loss can be measured based on time domain loss and frequency domain loss. For example, the reconstruction loss may be a negative sign-to-noise ratio (SNR) and L1 norm in log magnitutes of a short-time Fourier tansform (STFT). The reconstruction loss can be expressed as a weighted sum with weights (e.g., blending weights) as shown in Equation 12.

[Equation 12]

Quantized feature vectors ( , The entropy (e.g., empirical entropy) of ) can be calculated by observing the assignment frequency of each centroid to multiple feature vectors. As a result, the empirical entropy of the coding system The entropy estimate can be calculated as shown in Equation 13.

[Equation 13]

In equation 12, may represent the assignment probability for centroid j.

Entropy estimate ( ) bitrate counterpart ( ), to convert to the lower bound of the code dimension ( ) and frame rate ( ) can be considered as in Equation 14 below.

[Equation 14]

The bit rate of the coding system can be normalized by band-specific entropy loss. Entropy loss for each band can be calculated as Equation 15.

[Equation 15]

In equation 13, represents the bitrate estimated from the band-specific code vector, represents the Mokpo bitrate (band-specific target bitrate) for each band, May represent a weight for controlling the contribution of entropy loss to network loss.

Figure 4 is a flowchart for explaining the operation of an encoder according to an embodiment.

Referring to FIG. 4, according to one embodiment, operations 410 to 430 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel. Operations 410 to 430 may be substantially the same as the operations of the encoder (e.g., encoder 110 in FIG. 1 and encoder 310 in FIG. 3) described with reference to FIGS. 1 to 3. Accordingly, detailed description will be omitted.

In operation 410, the encoders 110 and 310 may acquire a full-band input signal (e.g., the full-band input signal of FIG. 3).

In operation 420, the encoders 110 and 310 convert the full-band input signal into a first sub-band signal using an encoder neural network (e.g., encoder neural network 311 in FIG. 3) including a plurality of encoding layers. A second feature vector corresponding to the feature vector and the second subband signal may be extracted.

At operation 430, the encoder 110, 310 compresses the first feature vector and the second feature vector to generate a first code vector corresponding to the first feature vector and a second code vector corresponding to the second feature vector. You can.

At operation 440, the encoders 110 and 310 may generate a bitstream by quantizing the first code vector and the second code vector.

Figure 5 is a flowchart for explaining the operation of a decoder according to an embodiment.

Referring to FIG. 5, according to one embodiment, operations 510 to 520 may be performed sequentially, but are not limited thereto. For example, operation 530 may be performed later than operation 520, or operations 520 to 530 may be performed in parallel. Operations 510 to 540 may be substantially the same as the operations of the decoder (e.g., the decoder 160 of FIG. 1 and the decoder 360 of FIG. 3) described with reference to FIGS. 1 to 3. Accordingly, detailed description will be omitted.

At operation 510, the decoder 160, 360 generates a reconstructed first feature vector corresponding to a first subband signal of the original full-band signal and a second subband of the original full-band signal. A restored second feature vector corresponding to the inverse signal can be obtained.

In operation 520, the decoders 160 and 360 may reconstruct the second subband signal based on the reconstructed second feature vector.

In operation 530, the decoders 160 and 360 may reconstruct the first subband signal based on the reconstructed first feature vector and the reconstructed second feature vector.

In operation 540, the decoders 160 and 360 may restore the original full-band signal based on the restored first sub-band signal and the restored second sub-band signal.

Figure 6 is a schematic block diagram of an encoder according to one embodiment.

Referring to FIG. 6, according to one embodiment, the encoder 600 (e.g., the encoder 110 of FIG. 1, the encoder 210 of FIG. 2, and the encoder 310 of FIG. 3) includes a memory 640 and a processor. It may include (620).

The memory 640 may store instructions (or programs) that can be executed by the processor 620. For example, the instructions may include instructions for executing the operation of the processor 620 and/or the operation of each component of the processor 620.

Memory 640 may include one or more computer-readable storage media. The memory 640 includes non-volatile storage elements (e.g., magnetic hard disk, optical disk, floppy disk, flash memory, electrically programmable memories (EPROM), It may include electrically erasable and programmable (EEPROM).

Memory 640 may be non-transitory media. The term “non-transitory” may indicate that the storage medium is not implemented as a carrier wave or propagated signal. However, the term “non-transitory” should not be interpreted as meaning that the memory 640 is immovable.

The processor 620 may process data stored in the memory 640. The processor 620 may execute computer-readable code (e.g., software) stored in the memory 640 and instructions triggered by the processor 620.

The processor 620 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

Operations performed by the processor 620 may be substantially the same as the operations of the encoders 110, 210, and 310 described with reference to FIGS. 1 to 4. Accordingly, detailed description will be omitted.

Figure 7 is a schematic block diagram of a decoder according to one embodiment.

Referring to FIG. 7, according to one embodiment, the decoder 700 (e.g., the decoder 160 of FIG. 1, the decoder 260 of FIG. 2, and the decoder 360 of FIG. 3) includes a memory 740 and a processor ( 720).

The memory 740 may store instructions (or programs) that can be executed by the processor 720. For example, the instructions may include instructions for executing the operation of the processor 720 and/or the operation of each component of the processor 720.

Memory 740 may include one or more computer-readable storage media. The memory 740 includes non-volatile storage elements (e.g., magnetic hard disk, optical disk, floppy disk, flash memory, electrically programmable memories (EPROM), It may include electrically erasable and programmable (EEPROM).

Memory 740 may be non-transitory media. The term “non-transitory” may indicate that the storage medium is not implemented as a carrier wave or propagated signal. However, the term “non-transitory” should not be interpreted as meaning that the memory 540 is immovable.

The processor 720 may process data stored in the memory 740. The processor 720 may execute computer-readable code (e.g., software) stored in the memory 740 and instructions triggered by the processor 720.

The processor 720 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

Operations performed by the processor 720 may be substantially the same as the operations of the decoders 160, 260, and 360 described with reference to FIGS. 1 to 3 and 5. Accordingly, detailed description will be omitted.

Figure 8 is a schematic block diagram of an electronic device according to an embodiment.

Referring to FIG. 8 , according to one embodiment, the electronic device 800 may include a memory 840 and a processor 820.

The memory 840 may store instructions (or programs) that can be executed by the processor 820. For example, the instructions may include instructions for executing the operation of the processor 820 and/or the operation of each component of the processor 820.

Memory 840 may include one or more computer-readable storage media. The memory 840 includes non-volatile storage elements (e.g., magnetic hard disk, optical disk, floppy disk, flash memory, electrically programmable memories (EPROM), It may include electrically erasable and programmable (EEPROM).

Memory 840 may be non-transitory media. The term “non-transitory” may indicate that the storage medium is not implemented as a carrier wave or propagated signal. However, the term “non-transitory” should not be interpreted as meaning that the memory 540 is immovable.

The processor 820 may process data stored in the memory 840. The processor 820 may execute computer-readable code (e.g., software) stored in the memory 840 and instructions triggered by the processor 820.

The processor 820 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The operations performed by the processor 820 include the encoder (e.g., encoder 110 in FIG. 1, encoder 210 in FIG. 2, encoder 310 in FIG. 3) and decoder (e.g., encoder 110 in FIG. 1, encoder 210 in FIG. 2, encoder 310 in FIG. 3) described with reference to FIGS. 1 to 5. Example: The operation of the decoder 160 in FIG. 1, the decoder 260 in FIG. 2, and the decoder 360 in FIG. 3 may be substantially the same. Accordingly, detailed description will be omitted.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be saved in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (19)

In the audio signal encoding method,
An operation to obtain a full-band input signal;
The full-band input signal is converted into a first feature vector corresponding to a first sub-band signal using an encoder neural network including a plurality of encoding layers. and extracting a second feature vector corresponding to the second subband signal;
An operation of compressing the first feature vector and the second feature vector to generate a first code vector corresponding to the first feature vector and a second code vector corresponding to the second feature vector. ; and
An operation of generating a bitstream by quantizing the first code vector and the second code vector.
Method, including.
According to paragraph 1,
The first subband signal is,
High band signal of the full-band input signal
Including,
The second subband signal is,
A downsampled core band signal of the full-band input signal.
Method, including.
According to paragraph 1,
Each of the plurality of encoding layers,
A method of encoding the output of a previous encoding layer.
According to paragraph 3,
The extracting operation is,
extracting an output of an intermediate encoding layer among the plurality of encoding layers as the first feature vector; and
An operation of extracting the output of the last encoding layer among the plurality of encoding layers as the second feature vector
method, including
According to paragraph 4,
Each of the plurality of encoding layers,
1 dimensional convolution layer
Method, including.
According to clause 5,
Among the plurality of encoding layers, the remaining encoding layers excluding one or more encoding layers located behind the middle encoding layer are:
With the same stride value,
The one or more encoding layers,
A method having a stride value greater than the remaining encoding layers.
According to clause 6,
The stride values of the remaining encoding layers are,
1 person, method.
In the audio signal decoding method,
A reconstructed first feature vector corresponding to a first subband signal of the original full-band signal and a reconstructed second feature vector corresponding to a second subband signal of the original full-band signal. An operation to obtain;
Restoring the second sub-band signal based on the restored second feature vector;
Restoring the first sub-band signal based on the restored first feature vector and the restored second feature vector; and
An operation of restoring the original full-band signal based on the restored first sub-band signal and the restored second sub-band signal.
Method, including.
According to clause 8,
The first subband signal is,
High-band signal of the original full-band signal
Including,
The second subband signal is,
Downsampled core band signal of the original full band signal
Method, including.
According to clause 8,
The operation of restoring the second sub-band signal is:
An operation of encoding the restored second feature vector using a decoder neural network including a plurality of decoding layers.
Including,
Each of the plurality of decoding radars,
A method of encoding the output of the previous decoding layer.
According to clause 8,
The operation of restoring the first subband signal includes:
An operation of encoding the reconstructed first feature vector and the reconstructed second feature vector using a decoder neural network including a plurality of decoding layers.
Including,
Each of the plurality of decoding layers,
Encode the output of the previous decoding layer,
Among the plurality of decoding layers, the intermediate decoding layer is:
A method of encoding the output of the previous decoding layer using the restored second feature vector.
In a device for encoding audio signals,
memory containing instructions; and
It is electrically connected to the memory and includes a processor for executing the instructions,
When the instructions are executed by the processor, the processor performs a plurality of operations,
The plurality of operations are,
An operation to obtain a full-band input signal;
The full-band input signal is converted into a first feature vector corresponding to a first sub-band signal using an encoder neural network including a plurality of encoding layers. and extracting a second feature vector corresponding to the second subband signal;
An operation of compressing the first feature vector and the second feature vector to generate a first code vector corresponding to the first feature vector and a second code vector corresponding to the second feature vector. ; and
An operation of generating a bitstream by quantizing the first code vector and the second code vector.
Device, including.
According to clause 12,
The first subband signal is,
High band signal of the full-band input signal
Including,
The second subband signal is,
A downsampled core band signal of the full-band input signal.
Device, including.
According to clause 12,
Each of the plurality of encoding layers,
A device that encodes the output of a previous encoding layer.
According to clause 14,
The extracting operation is,
extracting an output of an intermediate encoding layer among the plurality of encoding layers as the first feature vector; and
An operation of extracting the output of the last encoding layer among the plurality of encoding layers as the second feature vector
Device, including.
According to clause 15,
Each of the plurality of encoding layers,
1 dimensional convolution layer
Device, including.
According to clause 16,
Among the plurality of encoding layers, the remaining encoding layers excluding one or more encoding layers located behind the middle encoding layer are:
With the same stride value,
The one or more encoding layers,
A device having a stride value greater than the remaining encoding layers.
According to clause 17,
The stride values of the remaining encoding layers are,
1 person, device.
A computer program combined with hardware and stored in a computer-readable recording medium to execute the method of any one of claims 1 to 11.

Images