KR20220050458A - Lightweight Deep Learning Network Method and Device for 3D Human Model Reconstruction - Google Patents

Abstract

Provided is a method for determining an optimal quantization parameter, based on both a reconstructed image quality and compression performance, when lightening and compressing a deep learning network for reconstructing a human model. According to an embodiment of the present invention, a system for reconstructing a three-dimensional system includes: an image encoder to refine image feature vectors from an image input, through the deep learning network; a volume encoder to refine volume feature vectors from a depth image input through the deep learning network; and a volume decoder to reconstruct the three-dimensional model from the image feature vectors refined in the image encoder and the volume feature vectors refined in the volume encoder. In the deep network of the image encoder, the volume encoder, and the volume decoder, a weight may be quantized by adaptively applying multiple quantization manners.

Description

Lightweight Deep Learning Network Method and Device for 3D Human Model Reconstruction}

The present invention relates to a deep learning-based 3D human model restoration technology, and more particularly, to a network lightweight technology that maintains the human restoration performance of the existing network while alleviating the high power consumption problem of the existing deep learning reasoning machine.

1) Definition of deep learning network and the need for weight reduction

A deep learning network, also called kernel data, is composed of several layers, and each layer is composed of weights and biases.

Because one layer of kernel data has different weights and number of bias values depending on the layer type (Input / Output / Convolutional / Residual / Fully-Connected / Batch Normalization / Recurrent, etc.), the size of the kernel data varies according to the layer configuration.

Although the number of calculations varies depending on the layer type, as the length of the layer increases, the amount to be calculated increases. In general, a shorter layer is advantageous in terms of operation speed.

In addition, most of the deep learning operation precision is 32-bit floating point operation, and although operation with lower precision (eg 8-/16-bit, etc.) advantage in terms of

2) Conventional deep learning network lightweight technology

Attempts to reduce the accuracy of the network somewhat and reduce the weight of the model through compression techniques have also been proposed. It is a technology that compresses the VGG-16 network up to 49 times through the Pruning-Quantization-Huffman Encoding process.

The pruning technique determines that a value with a sufficiently small weight in the model does not contribute significantly to the performance, cuts the connection between neurons (the corresponding weight is set to 0), and maintains performance through the re-learning process. This is a method of removing unnecessary weights.

The quantization technique (quantization technique) derives an optimal quantization codebook for quantizing the weights in a layer in the learning process.

The last Huffman encoding technique is a process of converting quantized weights and codebooks (CodeBook, Index) into bitstreams, and additional compression of about 20 to 30% is performed in this process.

However, the prior art has not been able to adaptively use a quantization technique capable of bringing about further improvement in compression performance or to propose an entropy coding method suitable for distribution of input data.

The present invention has been devised to solve the above problems, and an object of the present invention is to lighten and compress a deep learning network for human model restoration, an optimal quantization parameter determination method that considers the restoration quality and compression performance at the same time. is in providing.

According to an embodiment of the present invention for achieving the above object, a three-dimensional model restoration system includes an image encoder for refining image feature vectors in an input image image using a deep learning network; a volume encoder that refines volume feature vectors in an input depth image using a deep learning network; A volume decoder that uses a deep learning network to reconstruct a 3D model from image feature vectors refined in the image encoder and volume feature vectors refined in the volume encoder; The learning network may be one in which weights are quantized by adaptively applying a plurality of quantization methods.

A deep learning network of an image encoder, a volume encoder, and a volume decoder can adaptively apply a quantization method for each layer.

The quantization method may include a Symmetric + Signed (SS) quantization method and an Asymmetric + unsigned (AU) quantization method.

It may be to apply a quantization method based on the mean, skewness, and kurtosis of the weight distribution.

If the skewness is within a specific range, the SS quantization method is applied, if the skewness is outside the specific range, the AU quantization method is applied, and the quantization bit can be determined based on the Kurtosis size.

A three-dimensional model restoration system according to the present invention comprises: an image feature vector compressor for compressing image feature vectors at a rear end of an image encoder; at the rear end of the volume encoder, a volume feature vector compressor for compressing the volume feature vectors; and an expander that expands the compressed image feature vectors and the volume feature vectors at the front end of the volume decoder.

The image feature vector compressor compresses the image feature vectors using any one of Huffman Coding (HFC) and Arithmetic Coding (ARC), and the volume feature vector compressor compresses the volume feature vectors using a run-length coding (RLC) technique. may be doing

According to another aspect of the present invention, using a deep learning network, refining image feature vectors from an input image image; refining volume feature vectors in an input depth image using a deep learning network; and reconstructing a three-dimensional model from the image feature vectors refined in the image encoder and the volume feature vectors refined in the volume encoder by using a deep learning network; A learning network is provided with a three-dimensional model reconstruction method, characterized in that weights are quantized by adaptively applying a plurality of quantization methods.

As described above, according to the embodiments of the present invention, power consumption can be reduced by providing a lightweight technique suitable for human model restoration, and the deep learning network compression effect can be increased by applying an optimal quantization technique, and selectively It is possible to increase the compression effect of the deep learning network through the use of entropy coding technique.

1 is a diagram showing a deep learning network for 3D human model restoration to which the present invention is applicable;
2 is a diagram showing range linear quantization methods;
3 is a diagram showing a deep learning network to which the quantization technique proposed in an embodiment of the present invention is applied;
4 is a diagram illustrating an additional application of an entropy coding technique to a 3D human model reconstruction network to which the quantization technique presented in an embodiment of the present invention is applied;
5 is a flowchart provided for explaining a 3D human model restoration method according to another embodiment of the present invention;
6 is a hardware structure of a 3D human model restoration apparatus according to another embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

Deep learning techniques are showing encouraging performance that surpasses existing traditional methods in various application fields such as image recognition, voice signal processing, and natural language processing.

The deep learning technique learns a network composed of weights and biases to suit a given application field, and then applies the learned network to the application field again to improve performance.

However, as the deep learning technology develops, the size of the deep learning network is gradually increasing, but the high power consumption of the deep learning inference machine is emerging as a problem to be applied to mobile applications where sustained use time is important.

Accordingly, in an embodiment of the present invention, in a deep learning network for 3D human model restoration, a network capable of maintaining the human restoration performance of the existing network while alleviating the problem of high power consumption of the existing deep learning reasoning machine is proposed.

Specifically, in an embodiment of the present invention, a lightweight method suitable for a network for human model restoration is provided, an adaptive quantization technique is applied for each layer when the quantization method is performed, and a different entropy coding technique is applied for each network.

1 is a diagram illustrating a deep learning network for 3D human model restoration to which the present invention is applicable. As shown, the deep learning network for 3D human model reconstruction is configured to include an image encoder (Image Encoder: 110), a volume encoder (Volume Encoder: 120), and a volume decoder (Volume Decoder: 130).

The image encoder 110 is a deep learning network that takes an RGB image as an input, passes it through an Auto Encoder network, and refines (compresses) it into a small amount of feature vectors compared to the input data.

The volume encoder 120 is a deep learning network that receives a Signed Distance Function (SDF) volume, which is 3D spatial information, as an input, and refines (compresses) it into a small amount of feature vectors. The SDF volume is created by projecting a depth image into a 3D space.

The volume decoder 130 is a deep learning network that decodes a 3D human model by inputting feature vectors output from the image encoder 110 and the volume encoder 120 .

The process of reconstructing the 3D human model is a process of arranging unnecessary data in the input image and depth image by the image encoder 110 and the volume encoder 120 and deriving only core feature vectors (encoding), using feature vectors. Thus, the volume decoder 130 restores 3D data (decoding).

The image encoder 110 and the volume encoder/decoder 120/130 are made of floating point arithmetic with 32-bit precision.

In the embodiment of the present invention, the weights of the image encoder 110 and the volume encoder/decoder 120/130 are quantized through a range linear quantization technique as shown in Equation (1) below. where s _w is the scale value, X _f is the input weight (32-bit), z _w is the zero point, and X _q is the output weight (quantized weight) (8-/16-bit).

X _q = s _w X _f - z _w (1)

The quantization process is divided into two methods as shown in FIG. 2 according to whether the value of z _w is 0 or not.

Symmetric + Signed (SS) quantization method is a method of mapping to 8-bit X _q with -128 ~ 127 based on the maximum value of 32-bit X _f , and z _w is 0. On the other hand, in the asymmetric + unsigned (AU) quantization method, z _w exists because it maps to 8-bit X _q having 0 to 255 considering the minimum and maximum values of X _f at the same time.

The SS quantization method has inferior quantization performance compared to the AU quantization method because it maps up to X _f values that are not actually used as X _q . On the other hand, since the AU quantization method cannot guarantee the range of the value of X _q + z _w to 8-bit depending on the range of the value of z _w in the inverse quantization process, there are more bits than the SS quantization method in the convolutional operation process. should use (The power consumption is relatively large.)

An embodiment of the present invention proposes a method for quantizing a network by adaptively using an SS quantization method and an AU quantization method. At this time, it is proposed to determine whether to perform the SS quantization method or the AU quantization method by looking at the distribution of the input X _f (mean, skewness: the degree of skewness of the distribution, kurtosis: the sharpness of the distribution).

If the distribution is skewed (Skewness > 0, left-skewed, < 0, right-skewed), the asymmetric technique is used, and when it is close to 0, the symmetric technique is used.

Also, in the case of a sharp distribution (when Kurtosis has a large value), quantization is performed with smaller bits instead of 8 bits, and in the case of a flat distribution, quantization is adaptively performed with high bits.

When the quantization method presented in the embodiment of the present invention is applied, it can be expressed as shown in FIG. 3 . In an embodiment of the present invention, a different type of quantization method may be performed for each layer.

In an embodiment of the present invention, after applying the quantization technique as shown in FIG. 3 , the output results of the image encoder 110 and the volume encoder 120 may be compressed by additionally applying an entropy coding technique.

FIG. 4 shows that an entropy coding technique is additionally applied to the 3D human model reconstruction network to which the quantization technique presented in the embodiment of the present invention is applied.

Since the SDF input of the volume encoder 120 is volume data composed of 0 and 1, additional compression is performed by applying run-length coding (RLC) to the result of the volume encoder 120 . Because RLC counts and records the length of 0 (run), compression performance is low in a situation where 0s do not appear consecutively, but compression performance is fast.

Huffman Coding (HFC), Arithmetic, which is slower than RLC because the result of the image encoder 110 contains major feature points (boundary, shape information, etc.) for RGB images, but shows high compression performance even in situations in which zero values are not continuous. Coding (ARC) is applied to secure compression performance.

Through this, the image encoder 110 and the volume encoder 120 may additionally perform compression.

Since the volume decoder 130 may be driven in a server or in a mobile terminal, additional compression of the output results of the image encoder 110 and the volume encoder 120 is connected to secure power performance from the viewpoint of the terminal. (Power consumption of accessing an external memory and fetching data consumes more than 10 times more power than the cost of decoding compression.) At the front end of the volume decoder 130, an HFC / ARC / RLC decoder for decoding compressed feature vectors should take precedence

5 is a flowchart provided to explain a 3D human model reconstruction method according to another embodiment of the present invention.

As shown, first, the image encoder 110 refines the feature vectors in the input RGB image image (S210), and the compressor provided at the rear end of the image encoder 110 compresses the image feature vectors refined in step S210 ( S220).

In the deep learning network implemented in the image encoder 110 , the SS quantization method and the AU quantization method are adaptively applied based on the mean, skewness, and kurtosis of the weight distribution, and the weights are quantized. This is the same for the volume encoder 120 and the volume decoder 130 below. In step S220, the HFC/ARC technique is applied.

The volume encoder 120 receives the SDF volume containing the depth image information and refines the feature vectors (S230), and the compressor provided at the rear end of the volume encoder 120 compresses the volume feature vectors refined in step S230 (S240). ). In step S240, the RLC technique is applied.

The decompressor provided at the rear end of the volume decoder 130 decompresses the image feature vectors compressed in step S220 and the volume feature vectors compressed in step S230 ( S250 ).

The volume decoder 130 restores the 3D human model by inputting the image feature vectors and the volume feature vectors extended in step S250 (S260).

6 is a hardware structure of a 3D human model restoration apparatus according to another embodiment of the present invention.

The apparatus for restoring a three-dimensional human model according to an embodiment of the present invention can be implemented as a computing system including a communication unit 310 , a processor 320 and a storage unit 330 , as shown.

The communication unit 310 is a means for communicating with an external device/network, and the processor 320 is a CPU and GPU for driving a deep learning network for 3D human model restoration shown in FIGS. 1, 3 and 4 . is a set The storage unit 330 provides a storage space necessary for the processor 320 to operate.

So far, a preferred embodiment has been described in detail for a lightweight deep learning network method and apparatus for 3D human model restoration.

In the above embodiment, the weight quantization technique was adaptively applied to lighten the deep learning network for human model restoration, and an appropriate compression technique was applied according to the characteristics of the feature vectors.

By reducing the weight and compression of the deep learning network suitable for human model restoration, power saving and compression effect can be expected.

On the other hand, it goes without saying that the technical idea of the present invention can be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical ideas according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium may be any data storage device readable by the computer and capable of storing data. For example, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

110: Image Encoder
120: Volume Encoder
130: Volume Decoder

Claims (8)

an image encoder for refining image feature vectors in an input image image using a deep learning network;
a volume encoder that refines volume feature vectors in an input depth image using a deep learning network;
A volume decoder that uses a deep learning network to reconstruct a 3D model from the image feature vectors refined in the image encoder and the volume feature vectors refined in the volume encoder;
A deep learning network of image encoders, volume encoders and volume decoders,
A three-dimensional model restoration system, characterized in that weights are quantized by adaptively applying a plurality of quantization methods.
The method according to claim 1,
A deep learning network of image encoders, volume encoders and volume decoders,
A three-dimensional model restoration system, characterized in that adaptively applying a quantization method for each layer.
3. The method according to claim 2,
The quantization method is
A three-dimensional model reconstruction system comprising a Symmetric + Signed (SS) quantization method and an Asymmetric + unsigned (AU) quantization method.
4. The method according to claim 3,
A three-dimensional model restoration system, characterized by applying a quantization method based on the mean, skewness, and kurtosis of the weight distribution.
5. The method according to claim 4,
If the skewness is within a specific range, the SS quantization method is applied, and if the skewness is outside the specific range, the AU quantization method is applied.
3D model reconstruction system, characterized in that the quantization bit is determined based on the size of Kurtosis.
The method according to claim 1,
at the rear end of the image encoder, an image feature vector compressor for compressing image feature vectors;
at the rear end of the volume encoder, a volume feature vector compressor for compressing the volume feature vectors; and
3D model reconstruction system, characterized in that it further comprises; an expander that expands the compressed image feature vectors and the volume feature vectors in the front end of the volume decoder.
7. The method of claim 6,
Image Features Vector Compressor,
Image feature vectors are compressed by any one of Huffman Coding (HFC) and Arithmetic Coding (ARC),
Volume feature vector compressor,
A 3D model reconstruction system, characterized in that the volume feature vectors are compressed using a run-length coding (RLC) technique.
refining image feature vectors from an input image using a deep learning network;
refining volume feature vectors in an input depth image using a deep learning network; and
Using a deep learning network, reconstructing a three-dimensional model from image feature vectors refined in an image encoder and volume feature vectors refined in a volume encoder;
A deep learning network of image encoders, volume encoders and volume decoders,
A 3D model reconstruction method, characterized in that weights are quantized by adaptively applying a plurality of quantization methods.

Images