KR20210033852A - Method and Apparatus for Artificial Band Conversion Based on Learning Model - Google Patents

Abstract

Disclosed are an artificial band conversion device and a method based on a learning model. To transform the low-quality sound source data compressed at a high compression rate and transmitted to a mobile device into a signal equivalent to a high-quality sound source due to the limitation of a transmission channel, the artificial band conversion device and method using an energy estimation method based on a learning model is provided.

Description

{Method and Apparatus for Artificial Band Conversion Based on Learning Model}

The present invention relates to an apparatus and method for converting an artificial band based on a learning model used in a mobile device.

The contents described below merely provide background information related to the present invention and do not constitute the prior art.

Artificial band conversion is a high frequency band based on analysis of a low frequency band (hereinafter referred to as "low band") for a low sound quality signal received at a low transmission rate due to a limited bandwidth or hardware performance. Hereinafter, it is also called ABE (Artificial Bandwidth Extension) as a technique for improving signal quality by generating "high band"). In the field of music sound signal processing, ABE is applied to a codec system collectively referred to as an encoder and decoder for data compression, and is operated on mobile devices, mainly in the form of additional post-processing to improve quality in online situations. Is executed as

As a conventional technology implementing the ABE, there is a method of separately compressing high-band information and transmitting it to a mobile device as in G.729.1 (see Non-Patent Document 1). Although there is an advantage of using high-band information for bandwidth extension, there is a disadvantage in that a bandwidth for additional bit allocation is required for a transmission channel, and the complexity of the codec used is increased.

Other technologies include an estimation method based on pattern recognition such as Hidden Markov Model (HMM) and Gaussian Mixture Model (GMM). However, the pattern recognition-based estimation method requires a long time to train the model to achieve the target performance, and the inference process applying the learning result is also complex. Despite the recent development of SoC (System on Chip) technology, real-time processing conditions for inference processes have improved compared to before, the pattern recognition-based estimation method has the disadvantage that real-time processing in mobile devices is difficult.

Therefore, while real-time processing is possible on a mobile device without burdening the transmission channel, an artificial band converter and method for processing a sound source signal with improved performance represented by a signal to noise ratio (SNR) are required.

Non-Patent Document 1: ITU-T (January 2007). "G.729: Coding of speech at 8 kbit/s using conjugate-structure algebraic-code-excited linear prediction (CS-ACELP)"

The present disclosure is an artificial band converter using an energy estimation method based on a learning model to transform low-quality sound source data compressed at a high compression rate and transmitted to a mobile device into a signal equivalent to a high-quality sound source due to a transmission channel constraint, and The main purpose is to provide a method.

According to an embodiment of the present invention, a characteristic extracting unit for extracting low frequency band energy for each sub-band by acquiring data in a frequency domain representing a low-quality signal; A band energy estimating unit including a pre-trained learning-type calculation model, and estimating a high frequency band energy by inputting the low-band energy into the learning-type calculation model; And a reconstruction unit estimating full frequency band frequency data using the high-band energy, the low-band energy, and data in the frequency domain.

According to another embodiment of the present invention, in the artificial band conversion method of the artificial band converter, the process of extracting low frequency band energy for each sub-band by using data in the frequency domain: the Estimating a high frequency band by inputting low-band energy into a pre-trained learning model; Generating normalized full frequency band data by using the low-band energy and data in the frequency domain; And generating full-band energy by synthesizing the high-band energy and the low-band energy, and then processing the leveled full-band data and the full-band energy to estimate full-band frequency data. Provides an implemented artificial band conversion method.

According to another embodiment of the present invention, there is provided a computer program stored in a computer-readable, nonvolatile or non-transitory recording medium in order to execute each step of the artificial band conversion method.

As described above, according to the present embodiment, in order to transform low-quality sound source data compressed at a high compression rate and transmitted into a high-quality sound source signal due to a limitation of a transmission channel, an artificial band converter using an energy estimation method based on a learning model. And by providing a method, there is an effect of increasing user satisfaction by providing sound quality equivalent to a high-quality sound source in a mobile device .

1 is a block diagram of an artificial band converter implemented on a mobile device according to an embodiment of the present invention.
2 is a block diagram of a bandwidth extension unit according to an embodiment of the present invention.
3 is a structural diagram of a learning model for estimating high-band energy according to an embodiment of the present invention.
4 is a flowchart of an artificial band conversion method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to elements of each drawing, it should be noted that the same elements are assigned the same numerals as possible, even if they are indicated on different drawings. In addition, in describing the embodiments, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the embodiments, a detailed description thereof will be omitted.

In addition, terms such as first, second, A, B, (a), (b) may be used to describe the constituent elements of the present embodiments. These terms are for distinguishing the constituent element from other constituent elements, and the nature, order, or order of the constituent element is not limited by the term. Throughout the specification, when a part'includes' or'includes' a certain element, it means that other elements may be further included rather than excluding other elements unless otherwise stated. . In addition, the'... Terms such as'sub' and'module' mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software.

DETAILED DESCRIPTION OF THE INVENTION The detailed description to be disclosed below together with the accompanying drawings is intended to describe exemplary embodiments of the present invention, and is not intended to represent the only embodiments in which the present invention may be practiced.

In the following description, it is assumed that the quality of the sound source is inversely proportional to the extrusion rate of the sound source (expressed in kbps), that is, it is assumed that the higher the compression rate, the lower the quality of the sound source. In addition, it is assumed that a low-quality sound source is relatively low frequency band data for a high-quality sound source.

1 is a block diagram of an artificial band converter implemented on a mobile device according to an embodiment of the present invention.

The artificial band converter 110 illustrated in FIG. 1 includes all or part of an input unit 111, a bandwidth extension unit 112, and an output unit 113. Here, the components included in the artificial band converter 110 according to the present embodiment are not necessarily limited thereto. For example, artificial band converter 110 On the top, a training unit (not shown) for training a learning model may be additionally provided, or may be implemented in a form interlocked with an external training unit.

The input unit 111 acquires necessary data in a process of performing artificial bandwidth extension (ABE) and converts it into a form suitable for an ABE.

For example, the input unit 111 according to the present embodiment receives the transmitted high compression rate sound source from the streaming service system and then transforms it into frequency domain data. Modified Discrete Cosine Transform (MDCT), Fast Fourier Transform (FFT), or cepstrum transformation may be used as the transformation method, but is not limited thereto.

를 생성한다. In this embodiment, the input unit 111 first synthesizes data in a time domain using a codec after receiving the sound source streaming data having a high compression rate. When synthesizing time domain data, it is assumed that a sampling frequency of 44.1 KHz is used. Next, with respect to the sound source data in the time domain, MDCT is performed while overlapping and sliding the execution section, and the data in the frequency domain

Create

For example, in the present embodiment, a sound source having a high compression rate refers to a sound source compressed by one of 64 kbps, 96 kbps, and 128 kbps, but is not limited thereto. The high-quality sound source targeted by ABE refers to a sound source of 320 kbps or equivalent quality, but is not limited thereto. It has a lower compression rate than a sound source with a high compression rate, and is one of the sound sources with the best sound quality that can be provided by streaming services. I can.

In addition, the input unit 111 according to another embodiment of the present invention may acquire low-quality sound source data that already exists in the form of data in the time domain.

The bandwidth extension unit 112 performs artificial band conversion on a low-quality sound source.

The bandwidth expansion unit 112 according to the present embodiment includes a characteristic extracting unit 121, a band energy estimating unit 122, and a reconstructing unit 123. The bandwidth extension unit 112 receives the data in the frequency domain converted by the input unit 111 as an input, and then converts the data into the frequency domain by executing an ABE. The structure and operation of the bandwidth extension unit 112 will be described later.

The output unit 113 provides the converted sound source to the user of the artificial band converter 110 in an audible form.

The output unit 113 according to the present exemplary embodiment receives the modified data in the frequency domain from the bandwidth extension unit 112 and converts it into an extended sound source in the time domain through a synthesis process. The synthesis process may include a process of converting data in the frequency domain into data in the time domain, such as inverse MDCT (IMDCT), inverse FFT (IFFT), or inverse cepstrum transform. Finally, the extended data in the time domain is output in an audible form.

The artificial band converter 110 according to the present embodiment may be implemented on the mobile device 100. The mobile device 100 may be any device capable of receiving a sound source streaming service using a wireless or wired transmission path that is limited by a transmission band.

2 is a block diagram of a bandwidth extension unit according to an embodiment of the present invention.

The bandwidth extension unit 112 performs artificial band conversion on a low-quality sound source. The bandwidth extension unit 112 includes all or part of the characteristic extracting unit 121, the band energy estimating unit 122, and the reconfiguration unit 123. The reconstruction unit 123 includes a normalization unit 201, a frequency band replication unit 202 (Spectral Band Replication: SBR), an energy synthesis unit 203, and a full-band data estimation unit 204, but is not limited thereto. . 2 is an exemplary configuration according to the present embodiment, and implementation including other components or other connections between components is possible based on a data processing technique in the frequency domain.

로 표현된 음원 신호를 입력으로 받아들여 부대역(sub-band) 별로 저대역(low frequency band) 에너지를 추출한다. 추출하는 방법은 수학식 1에 따른다. 수학식 1에 따르면 하나의 부대역 에너지를 구하기 위해 SB의 2 배수에 해당하는 주파수 영역의 데이터를 이용하고, 각 부대역은 서로 50 %씩 겹쳐져(overlapping) 있다. 전술한 바와 같은 부대역 에너지를 구하는 방법은 하나의 예시이며, SB 및 수학식 1을 변형하여 다양한 방법으로 부대역 에너지를 구하는 것이 가능하다.The characteristic extraction unit 121 is converted by the input unit 111 Frequency domain data

The sound source signal represented by is received as an input and low frequency band energy is extracted for each sub-band. The extraction method is according to Equation 1. According to Equation 1, data in a frequency domain corresponding to a multiple of 2 SB is used to obtain one subband energy, and each subband overlaps each other by 50%. The method of obtaining the subband energy as described above is an example, and it is possible to obtain the subband energy in various ways by modifying the SB and Equation 1.

를 구성요소로 포함하는 저대역 에너지 벡터

은 대역에너지 추정부(122)에 입력으로 전달된다. 따라서

은

개의 부대역 에너지를 포함하고,

은 확장 대상 음원의 압축률에 의존한다.

Low-band energy vector containing as components

Is transmitted to the band energy estimation unit 122 as an input. therefore

silver

Contains subband energies,

Depends on the compression rate of the sound source to be expanded.

를 처리하여 축소된 차원의 데이터인 저대역 에너지 벡터를 대역에너지 추정부(122)에 제공함으로써, 대역에너지 추정부(122) 구현 시 복잡도를 감소시키고, 감소된 복잡도에 따라 모바일 디바이스(100) 상에서의 실시간 동작 가능성 및 실시간 동작 성능을 높일 수 있다.As described above, in this embodiment, data in the frequency domain

By processing and providing the reduced-dimensional data, the low-band energy vector, to the band energy estimating unit 122, the complexity of implementing the band energy estimating unit 122 is reduced, and on the mobile device 100 according to the reduced complexity. It is possible to increase the real-time operation possibility and real-time operation performance.

을 입력으로 받아들여 고대역(high frequency band) 에너지

를 추정한다. 대역에너지 추정부(122)는 저대역 에너지를 고대역 에너지 추정에 직접적으로 이용하기 때문에 주파수 영역 상의 음원 특성을 더 강조하는 고대역 에너지 추정이 가능하다. 대역에너지 추정부(122)는 고대역 에너지 추정을 위하여 트레이닝부에 의하여 기 학습된 신경회로망(Neural Network) 기반의 학습 모델을 이용한다. 학습 모델의 구조 및 학습 모델의 트레이닝 과정은 추후에 설명하기로 한다.The band energy estimating unit 122 is a low band energy

Is taken as an input and the high frequency band energy

Estimate Since the band energy estimating unit 122 directly uses the low band energy to estimate the high band energy, it is possible to estimate the high band energy that further emphasizes the sound source characteristics in the frequency domain. The band energy estimating unit 122 uses a learning model based on a neural network previously learned by the training unit to estimate high band energy. The structure of the learning model and the training process of the learning model will be described later.

를 이용하여 전대역(full frequency band) 주파수 데이터를 추정한다. The reconstruction unit 123 includes high-band energy estimated by the band energy estimation unit 122, low-band energy extracted by the characteristic extraction unit 121, and data in the frequency domain.

Estimate full frequency band frequency data using.

The normalization unit 201 generates leveled low-band data by processing the low-band energy and frequency-domain data extracted for each sub-band. The reason for applying the leveling is that it is possible to reduce a deviation that occurs when the frequency band replicating unit 202 obtains a correlation.

를 생성한다.The frequency band replicating unit 202 restores high band data by using the leveled low band data. In the restoration process, restoration is performed in the order of subbands, and the next (high frequency direction) subband of the subband restored to the present is the restoration target. Of the subband to be restored After just found a bag in front of the station and the correlation data with a high degree of low frequency region, thereby copying the data in the found area to target restored sub-band. Here, the subband immediately preceding is the last subband within the low band in the first stage of the restoration process, and the subband restored immediately before in the subsequent stages. The frequency band replicating unit 202 synthesizes the restored high-band data and the leveled low-band data to obtain leveled full-band data.

Create

와 특성추출부(121)에서 추출한 저대역 에너지

을 합성하여 전대역 에너지

를 생성한다.On the other hand, the energy synthesis unit 203 is the high-band energy estimated by the band energy estimation unit 122

And low-band energy extracted from the characteristic extraction unit 121

By synthesizing the full-band energy

Create

과 전대역 에너지

를 곱하여 전대역 주파수 데이터

를 추정한다. 주파수 영역에서의 곱셈을 수행하므로, 시간 영역에서의 콘볼루션(convolution) 과정을 수행하는 것과 동일하다. 따라서 곱셈 과정은 평준화된 주파수 데이터를 부대역 별 에너지에 해당하는 계수를 갖는 필터를 이용하여 필터링하는 것과 동일한 과정이며, 달리 말하면 평준화된 전대역 데이터를 전대역 에너지로 마스킹(masking)하는 과정이다. The full-band data estimating unit 204 is standardized full-band data

And full-band energy

Multiply by the full-band frequency data

Estimate Since multiplication is performed in the frequency domain, it is the same as performing a convolution process in the time domain. Therefore, the multiplication process is the same process as filtering the leveled frequency data using a filter having a coefficient corresponding to the energy of each subband, in other words, a process of masking the leveled full-band data with full-band energy.

3 is a structural diagram of a learning model for estimating high-band energy according to an embodiment of the present invention.

The learning model shown in FIG. 3 is based on a deep learning model. Hereinafter, a structure and a learning process of the deep learning model according to the present embodiment will be described with reference to FIG . 3.

The deep learning model includes an input layer, three fully-connected layers (fully-connected layer or dense layer, hereinafter referred to as a first fully-connected layer, a second fully-connected layer, and a third fully-connected layer), and an output layer. ), but is not necessarily limited thereto, and the number of nodes constituting each layer according to the time required for training, the performance of the band energy estimator 122, and the implemented implementation possibility, and the activation function for the output of the node ( activation function) and the number of all connection layers.

)만큼의 노드(node)로 구성되는 한편, 출력 레이어는 31 개의 부대역 에너지에 해당하는 31 개(

)의 노드를 포함한다. 전연결 레이어는 연결되는 이전 레이어와 전연결(fully-connected)된다. 따라서, 도 3의 도시에서 입력 레이어는 이어지는 제1 및 제3 전연결 레이어와 전연결되나, 제2 및 제3 전연결 레이어의 출력은 뒷단의 출력 레이어에서 단순히 합성된다. The deep learning model according to this embodiment outputs 31 subband energies for 11, 16 and 23 subband energy inputs, respectively, according to sound source compression rates of 64k bps, 96 kbps and 128 kbps. Thus, the input layer has a number corresponding to 11, 16 and 23 subband energy inputs (

), while the output layer is composed of 31 subband energies.

) Of the node. The fully-connected layer is fully-connected with the previous layer to which it is connected. Accordingly, in the illustration of FIG. 3, the input layer is pre-connected with the subsequent first and third pre-connected layers, but the outputs of the second and third pre-connected layers are simply synthesized in the output layer at the rear end.

The first full connection layer acts as a hidden layer, includes 64 nodes, and is fully connected to the second full connection layer including 31 nodes. The first and second all-connected layers are connected in series to each other to estimate high-band energy. Meanwhile, the third pre-connected layer includes the same number of nodes as the output layer and connects the input layer and the output layer in parallel with the first and second pre-connected layers. Accordingly, the third pre-connected layer assists in recursively estimating high-band energy by forming a shortcut path and transferring information of low-band energy, which is highly correlated with high-band energy, to the output side.

The operation of the deep learning model as described above can be expressed by Equation 2.

이고,

는 i 레이어와 j 레이 어 사이의 전연결 가중치 행렬(weight matrix),

는 i 레이어와 j 레이어 사이의 편향치 벡터(bias vector),

는 k 레이어의 출력 벡터를 나타낸다. Where the input vector x is the low-band energy

ego,

Is the total connection weight matrix between the i layer and the j layer,

Is the bias vector between layer i and layer j,

Denotes the output vector of the k-layer.

를 생성한다. 여기서 전처리 과정은 압축된 음원을 시간 영역 상의 데이터로 합성하는 과정, 시간 영역의 데이터를 주파수 영역의 데이터로 변환하는 과정, 및 주파수 영역의 데이터를 이용하여 부대역 별 대역 에너지를 추출하는 과정 등을 포함한다. Hereinafter, a training process of the deep learning model by the training unit (not shown) will be described. First, target data by applying the preprocessing process according to the present invention to a high-quality sound source (eg, a sound source compressed to 320 kbps)

Create Here, the pre-processing process includes a process of synthesizing the compressed sound source into time domain data, a process of converting time domain data into frequency domain data, and a process of extracting band energy for each subband using the frequency domain data. Includes.

을 대역에너지 추정부(122)에 입력하여 전대역 에너지 출력

를 산정한다. 트레이닝부는 타겟 데이터

와 딥러닝 모델의 출력

간의 거리 메트릭에 기반하여 딥러닝 모델의 파라미터를 업데이트한다. 여기서 거리 메트릭은 L1 및 L2 메트릭 등, 두 비교 대상 간의 메트릭 거리 차이를 표현할 수 있는 것이면 어느 것이든 이용 가능하다.Next, the low-band energy of a low-quality learning sound source (e.g., a sound source compressed to 64 kbps, 96 kbps or 128 kbps) corresponding to a high-quality sound source

Input to the band energy estimating unit 122 to output full band energy

Calculate Training department target data

And the output of the deep learning model

The parameters of the deep learning model are updated based on the distance metric between them. Here, any distance metric, such as an L1 and an L2 metric, can be used as long as it can express the difference in the metric distance between the two comparison targets.

In order to increase the training efficiency for the deep learning model, the energy representing the value of each node is used by applying a log function. Since the logarithmic function is applied to the energy, negative values may appear. Therefore, in order to properly reflect the influence of negative values in the training process, the active function Exponential Linear Unit (ELU) is applied to the output of the all-connected layer as shown in FIG. 3.

로 표기되어 있다. 그 이유는 트레이닝 과정이 아닌 대역폭확대부(112)의 일부로 동작 시, 딥러닝 모델이 고대역 에너지를 추정하는 것을 표현하기 위함이다. 전술한 바와 같이 트레이닝 과정에서는 전대역에 대한 타겟 데이터가 존재하므로 전대역 에너지를 추정하도록 트레이닝하는 것이 가능하다. 그러나 대역폭확대부(112)의 일부로 동작 시, 딥러닝 모델의 입력으로 저대역 에너지가 추출되므로, 추출된 저대역 에너지를 재활용하는 것이 합리적이다. 따라서 딥러닝 모델이 대역폭확대부(112)의 일부로 동작 시에는 딥러닝 모델이 구한 전대역 에너지 중, 고대역 에너지에 해당하는 부분만을 사용한다. 2 and Equation 2, the output of the deep learning model is not full-band energy, but high-band energy

It is marked as. The reason is to express that the deep learning model estimates high-band energy when operating as a part of the bandwidth expansion unit 112 rather than a training process. As described above, in the training process, since target data for the full band exists, it is possible to train to estimate the full band energy. However, when operating as a part of the bandwidth expansion unit 112, since low-band energy is extracted as an input of the deep learning model, it is reasonable to recycle the extracted low-band energy. Therefore, when the deep learning model operates as a part of the bandwidth expansion unit 112, only a portion corresponding to the high-band energy is used among the full-band energy obtained by the deep learning model.

The mobile device 100 on which the artificial band converter 110 according to the present embodiment is mounted may be a programmable computer and includes at least one communication interface capable of being connected to a server (not shown).

Training on the learning model as described above may be performed in the server. The training unit of the server may perform training on the deep learning model having the same structure as the learning model mounted on the mobile device 100. Using a communication interface connected to the mobile device 100, the server transmits the trained parameters to the mobile device 100, and the artificial band converter 110 may update the parameters of the learning model using the received parameters. . In addition, when the mobile device 100 is shipped or when the artificial band converter 110 is mounted on the mobile device 100, a parameter of the learning model may be set.

4 is a flowchart of an artificial band conversion method according to an embodiment of the present invention.

In the flowchart illustrated in FIG. 4, the process performed by the input unit 111 and the output unit 113 as illustrated in FIG. 1 is omitted to mainly express the artificial band conversion method performed by the bandwidth expansion unit 112. In addition, in describing each process in the flow chart, a description of the components of the bandwidth expansion unit 112 performing each process is also covered in the description of FIG. 2, and thus further detailed description thereof will be omitted.

First, the low-band energy is extracted for each sub-band using the data in the frequency domain (S401), and the high-band energy is estimated by inputting the low-band energy into the band energy estimating unit 122 (S402), and then the low-band energy And processing the data in the frequency domain to generate leveled low-band data (S403).

Next, after restoring high-band data using the leveled low-band data, the restored high-band data and the leveled low-band data are synthesized to generate leveled full-band data (S404).

Next, after generating full-band energy by synthesizing the high-band energy estimated by the band energy estimating unit 122 and the low-band energy extracted from the characteristic extracting unit 121, the full-band frequency data is multiplied by the normalized full-band data and the full-band energy. It is estimated (S405).

Hereinafter, a result of evaluating the performance of the artificial band converter according to the present embodiment will be described. Compressed sound sources such as classical music, K-pop and pop music were used for evaluation and learning. After training the band energy estimating unit 122 according to the present embodiment by applying the above-described training method, a general HMM-based estimation method and SNR measurement results were compared using the same sound source for evaluation. The measured SNR is 20.8 dB, which shows better performance compared to the 17.5 dB measured in the HMM-based estimation method.

As described above, according to the present embodiment, in order to transform low-quality sound source data compressed at a high compression rate and transmitted into a high-quality sound source signal due to the limitation of a transmission channel, an artificial band converter using an energy estimation method based on a learning model. And by providing a method, there is an effect of increasing user satisfaction by providing sound quality equivalent to a high-quality sound source in a mobile device.

Each flow chart according to the present embodiment describes that each process is sequentially executed, but is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flow chart or execute one or more processes in parallel, the flow chart is not limited to a time series order.

Various implementations of the systems and techniques described herein include digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or their It can be realized in combination. Various such implementations may include being implemented as one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from and transmit data and instructions to and from a storage system, at least one input device, and at least one output device. Or a general purpose processor). Computer programs (which are also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable medium".

A computer-readable medium is any computer program product, apparatus, and/or device (e.g., CD-ROM, ROM, memory card, It represents a nonvolatile or non-transitory recording medium such as a hard disk, magneto-optical disk, and storage device).

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, nonvolatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, the programmable computer may be one of a server, a network device, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a personal data assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of the present embodiment, and those of ordinary skill in the technical field to which the present embodiment belongs will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain the technical idea, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

100: mobile device 110: artificial band converter
111: input unit 112: bandwidth extension unit
113: output unit 121: characteristic extraction unit
122: band energy estimation unit 123: reconstruction unit

Claims (13)

A characteristic extracting unit that obtains data in a frequency domain corresponding to a low-quality signal and extracts low frequency band energy for each sub-band;
A band energy estimating unit including a pre-trained learning-type calculation model, and estimating a high frequency band energy by inputting the low-band energy into the learning-type calculation model; And
Reconstruction unit for estimating high-quality full frequency band data using the high-band energy, the low-band energy, and data in the frequency domain
Artificial band converter comprising a. The method of claim 1,
The characteristic extracting unit,
An artificial band converter, characterized in that for extracting a reduced-dimensional low-band energy compared to the data in the frequency domain. The method of claim 1,
The learning-type computational model is a deep learning model, which is pre-trained using each of the high-quality full-band energy in the frequency domain and low-quality low-band energy in the frequency domain extracted from the same sound source signal as targets and inputs. Artificial band converter, characterized in that. The method of claim 3,
The deep learning model,
An artificial band converter, characterized in that the all-connected layers other than one of the plurality of all-connected layers are connected in series between the input layer and the output layer. The method of claim 4,
The deep learning model,
Wherein the one full connection layer connects the input layer and the output layer to transmit information on the low-band energy having a high correlation with the high-band energy to the output layer. The method of claim 4,
The deep learning model,
An artificial band converter, characterized in that an ELU (Exponential Linear Unit) is used as an activation function applied to the outputs of the plurality of all connection layers. The method of claim 1,
The reconstruction unit,
A normalization unit for generating leveled low-band data by processing the low-band energy and data in the frequency domain;
A frequency band replicating unit generating leveled full-band data by using the leveled low-band data;
An energy synthesis unit for synthesizing the high-band energy and the low-band energy to generate full-band energy; And
A full-band data estimating unit for estimating the high-quality full-band data by processing the leveled full-band data and the full-band energy
Artificial band converter comprising a. The method of claim 7,
The frequency band replicating unit,
An artificial band, characterized in that after restoring the high-band data based on a correlation between the leveled low-band data, and synthesizing the restored high-band data and the leveled low-band data to generate the leveled full-band data. Inverter. The method of claim 7,
The energy synthesis unit,
And generating the full-band energy by synthesizing the high-band energy estimated by the band energy estimating unit and the low-band energy extracted by the feature extracting unit. The method of claim 7,
The full-band data estimation unit,
By masking the normalized full-band data with the full-band energy using multiplication in the frequency domain, And estimating the high-quality full-band data. In the artificial band conversion method of the artificial band conversion device,
Process of extracting low frequency band energy for each sub-band using low quality frequency domain data:
Estimating a high frequency band by inputting the low band energy into a pre-trained learning model;
Generating normalized full frequency band data by using the low-band energy and data in the frequency domain; And
The process of estimating high-quality full-band data by synthesizing the high-band energy and the low-band energy to generate full-band energy, and then processing the leveled full-band data and the full-band energy
Characterized in that it comprises a, artificial band conversion method implemented on a computer. The method of claim 11,
The process of estimating the high-band energy,
An artificial band transformation method implemented on a computer, characterized by using the learning model that is pre-trained based on high-quality full-band energy in the frequency domain and low-quality low-band energy in the frequency domain extracted from the same sound source signal. A computer program stored in a computer-readable, nonvolatile or non-transitory recording medium to execute each step of the artificial band conversion method according to claims 11 and 12.

Images