AU2021105154A4

AU2021105154A4 - Adaptive Hierarchical Sampling for image Classification

Info

Publication number: AU2021105154A4
Application number: AU2021105154A
Authority: AU
Inventors: Zhenya Yue
Original assignee: Yunshigao Technology Co Ltd
Current assignee: Yunshigao Technology Co Ltd
Priority date: 2021-08-09
Filing date: 2021-08-09
Publication date: 2021-11-11
Anticipated expiration: 2029-08-09

Abstract

: Deterministic down-sampling of an unordered point cloud in a deep neural network has not been rigorously studied so far. Existing methods down-sample the points regardless of their importance to the network output. As a result, some important points in the point cloud may be removed, while less valuable points may be passed to next layers. In contrast, the proposed adaptive down-sampling method samples the points by taking into account the importance of each point, which varies according to application, task and training data. In this patent, we propose a novel deterministic, adaptive, permutation-invariant down-sampling layer, called Critical Points Layer (CPL), which learns to reduce the number of points in an unordered point cloud while retaining the important (critical) ones. Unlike most graph-based point cloud down-sampling methods that use k-NN to find the neighboring points, CPL is a global down-sampling method, rendering it computationally very efficient. The proposed layer can be used along with a graph-based point cloud convolution layer to form a convolutional neural network, dubbed CP-Net in this patent. We introduce a CP-Net for 3D object classification task that achieves high accuracy for the ModelNet40 dataset among point cloud based methods, which validates the effectiveness of the CPL. 1 FigI. features features points -___ __ Gather " " " resize Points smemsas max arg-max sort _sorto+ x dx I set Critical Points 11 1l 1l 11 | 1 1 I l I| || I 7 -L D I LEI ElMaximum value in column fS Fig2. features features poits resize Gather nPoints repeat max arg _max fsu a 1 st Weighted 1 1 11 11 11 11 11 11 _ ___ ___ U 1+!,y t Critical Points El Max imumnvalue in column 1"X" [21

Description

FigI. features

features points -___ __ Gather " "

" resize Points smemsas

max arg-max sort x _sorto+ dx I

set Critical Points I| 11 1l 1l 11| 1 1 I l || I7-LDILEI ElMaximum value in column fS

Fig2. features

features

poits resize Gather nPoints

repeat max arg _max fsu a

1 st Weighted 1 1 11 11 11 11 11 11 _ ___ ___ U1+!,y t Critical Points ElMax imumnvalue in column

1"X" [21

1. Background and Purpose In most robotic applications, laser point cloud data plays a key role in perception

of the surrounding environment. Autonomous mobile robotic systems in particular use

point cloud data to train deep models to solve different problems, such as dynamic

object detection, Simultaneous Localization and Mapping (SLAM), path planning, etc.

With the introduction of many new methods, such as PointNet, PointNet++, DGCNN,

PointCNN, and SONet, extracting features from unordered point cloud with deep

neural networks has become a highly active field of research. These methods are shown

to be quite successful in point cloud classification benchmarks, such as ModelNet40.

In practical scenarios, the number of points in the point cloud associated with an

object may be quite large, especially as a result of using high density sensors such as

Velodyne-64. One possible way to reduce computation is to down-sample the points in

the point cloud as it gets passed through the network. A class of methods are proposed

in which k-NN search is used to find the neighbourhood for each point and

down-sample according to these neighbourhoods. Such methods, however, trade one

kind of expensive computation (neighbourhood search) for another one (processing

large point cloud). In addition, deterministic down-sampling of an unordered point

cloud in a deep neural network has not been rigorously studied so far. Existing methods

down-sample the points regardless of their importance to the network output. As a

result, some important points in the point cloud may be removed, while less valuable

points may be passed to next layers. In contrast, the proposed adaptive down-sampling

method samples the points by taking into account the importance of each point, which

varies according to application, task and training data.

In this patent, we propose a novel deterministic, adaptive, permutation-invariant

down-sampling layer, called Critical Points Layer (CPL), which learns to reduce the

number of points in an unordered point cloud while retaining the important (critical)

ones. Unlike most graph-based point cloud down-sampling methods that use k-NN to

find the neighboring points, CPL is a global down-sampling method, rendering it

computationally very efficient. The proposed layer can be used along with a

graph-based point cloud convolution layer to form a convolutional neural network, dubbed CP-Net in this patent. We introduce a CP-Net for 3D object classification task that achieves high accuracy for the ModelNet40 dataset among point cloud based methods, which validates the effectiveness of the CPL.

2. New adaptive down-sampling Layers In this patent, we propose a novel deterministic, adaptive, permutation-invariant

find the neighboring points, CPL is a global down-sampling method, rendering it

computationally very efficient. The proposed layer can be used along with a

graph-based point cloud convolution layer to form a convolutional neural network,

dubbed CP-Net in this patent.

The new methods in our present invention will be described further in detail.

2.1. Employ Critical Points Layer (CPL) Lets assume the input to the CPL is an unordered point cloud with n points, each

represented as a feature vector xe Rd, where R is the set of real numbers and d is

the dimension of the feature vector. The goal of CPL is to generate a subset of input

points, called Critical Points (CP), with m < n points, each represented as a feature

vector ye R' , where 1 is the dimension of the new feature vector. The critical points of

a point cloud are the points with maximum information that are needed to be preserved

in a down-sampling (or pooling) process. These points may be changed based on the

task and application.

The block diagram of the proposed Critical Points Layer (CPL) is illustrated in

Figure la. The steps of the algorithm are explained in more details as follows:

Step 1: Get the maximumfeature value. The input point cloud Fs is a matrix with n rows (corresponding to n input points) and d columns (corresponding to d

-dimensional feature vectors). In the first step, the maximum feature value is obtained

for each column of the matrix Fs. This is the same as the max-pooling operation in

Point-Net. The resulting d-dimensional feature vector, denoted by finx, has the same

dimension as input feature vectors and can be independently used for classification and segmentation tasks. However, we are interested in down-sampling the input points rather than generating a single feature vector out of them. To this aim, the index of each row with a maximum feature value is also saved in the index vector idx. Vector idx contains the indices of all the points that have contributed to the feature vector fx. By definition, we call these points, the Critical Points (CP). These are the important points that should be preserved in the down-sampling process.

Step 2: Set the unique indices. Index vector idx may contain multiple instances of

the same point. To avoid these repetitions, unique indices are extracted from idx, using

the "set (unique)" function. Output set which has the unique indices is called the

Critical Set (CS) and is denoted by uidx. Beside finding the unique vector, we also add

up the feature values from inax that correspond to the same point or index. Resulting

feature vector fs will be later used to sort the input points.

Step 3: Sort the feature vector fs . Next, feature vector fs is sorted (in an

ascending order). Corresponding indices in uidx are also rearranged based on the

sorting output, resulting in an index vector which is denoted by suidx. This step is

necessary for the following sampling (resizing) operation. It also makes CPL invariant

to the order of input points.

Step 4: Resize and up-sample index vector suidx. Number of elements in suidx

may differ for different point clouds in the input batch. For batch processing however,

these numbers need to be the same. To address this, for each point cloud in the input

batch, the index vector suidx is up-sampled to a fixed size vector rsuidx using an

up-sampling method for integer arrays, such as nearest neighbor resizing.

Step 5: Gatherpointsand correspondingfeature.As the final step, the up-sampled

index vector rsuidx, which contains the indices of all the critical points, is used to

gather points and their corresponding feature vectors. Since different feature vectors

may correspond to a single point, and because of the information being filtered in

hidden NN layers, we may want to gather the features from other layers (denoted by F,)

than those used for selecting the points (denoted by Fs ). However, critical points are

defined based on the contribution of each point in the maximum feature vector obtained

from Fs, thus here we use F, = Fs.

2.2. Weighted Critical Points Layer (WCPL)

Step 1: Get the maximumfeature value and set the unique indices. The maximum

feature value is obtained for each column of the matrix Fs method the same as CPL's.

To avoid these repetitions, On the basis of what CPL does, WCPL add a vector f, that

records the number of contributions per point.

Step 2: Sort thefeature vector fs and repeat. Next, feature vector fs is sorted

and its corresponding indices in uidx are also rearranged based on the sorting output. To

increase the weight of a point by a factor of C, we repeat the point index C times. By

increasing the repetition frequency, the probability of selecting the point in the

down-sampling process will also increase. From another point of view, in WCPL, the

probability of missing a critical point in the output is lower than that in CPL. Then, for

each point cloud in the input batch, the index vector midx is resized to a fixed size

vector rsuidx. Step 3: Gatherpoints and correspondingfeature. As the final step, the resized

index vector rmidx, which contains the indices of all the critical points, is used to gather

points and their corresponding feature vectors.

3. Critical Points Net (CP-Net) In this patent, we propose a hierarchical architecture to apply deep convolutional

neural networks to point clouds, by systematically reducing the number of points using

the proposed CPL/WCPL. In the proposed network model, named Critical Points Net

(CP-Net), any graph convolution method can be used in convolution layers. The block

diagram of CP-Net using EdgeCony as an example is shown in Figure 2.

Step 1: Pass the point cloud into a convolution layer. The input in Figure 2 is an

unordered point cloud of size n. In the first step, the point cloud is passed into a

convolution layer of choice to filter the input into a richer set of features. In the

proposed network model, named Critical Points Net (CP-Net), any graph convolution

method, such as DCNN, GCNN, MoNet or EdgeCony (from DGCNN) can be used in

convolution layers. Step 2: Use the CPL/WCPL to down-sample the point cloud. The filtered output point cloud Fs. is then used as an input to thefirst CPL/WCPL. Using a CPL/WCPL with down-sampling factor ko , the number of points in Fs. is reduced to n / ko points. These steps are repeated for as many times as necessary to achieve a desired size of the point cloud (both in terms of number of points and feature vector size). Note that at j-th CPL/WCPL block, one can also benefit from using or concatenating features from all or some of the previous layers, i.e., {F F, --, F, }, as long as they correspond to the same points. As a result, the number of output points will be n

4. CP-Net for 3D Object Classification Here we give an example of CP-Net application in the 3D object classification problem. Block diagram of the proposed network is illustrated in Figure 3. The network is composed of three subnets: 1) n-point feature extraction subnet, 2) (n/4)-point subnet and 3) classification subnet. The detailed steps of the proposed network are as follows:

Network input is an unordered point cloud of size nx 3, where each point is a 3D vector. The input data goes through a spatial transformer network as explained in, to make it robust against any rigid transformation, including rotation and translation. It is worth noting that instead of using the original input, a modified version of EdgeConv edge feature is used for spatial transformation, as explained in the next step. The output of the spatial transform goes into a filtering CNN, here EdgeCony, to produce richer features. Unlike the original EdgeCony operator which uses two kernels in the edge feature function, we use the triple-kernel version ho (x,,x - x,,(x - x,)2), where (x - x,)2 is element-wise square operation between each point x, and its neighbouring point xi. In the proposed network, applying the EdgeCony with 128 filters to the input point cloud of size nx 3 , results in a point cloud of size nx 128. A multi-layer perceptron (MLP) layer expands the feature dimension from 128 to 1024 features, resulting in a point cloud of size nx 1024. Next, CPL/WCPL is applied to find the critical points and to reduce the number of input points. This step reduces the computational complexity without any loss in the classification accuracy. A down-sampling factor of 1/4 is chosen to reduce the number of points from n to n/4. Another EdgeConv layer is used to filter the point cloud, this time by preserving the depth and size to further process the received point cloud. Note that reducing the number of points in the previous layer highly reduces the computational complexity of the this layer. A reduce-max layer is used to generate a vector of size 1024, out of the point cloud of size nx 1024 . Finally, fully connected layers of size 512, 256 and 40 are applied to transform the feature vector of size 1024 to the number of classes in the

ModelNet40 dataset, which is 40. In the proposed 3D classification method, standard

softmax cross entropy is used as the loss function. In addition, all layers include a

ReLU activation function and batch normalization.

The classification accuracy results for our proposed CPNet/WCP-Net are shown in

Table 1 with comparisons against the previously proposed methods. As illustrated, our

CP-Net/WCP-Net methods rank as the runner-up to RS-CNN and surpass the accuracy

of all other methods in Table 1 and of ModelNet40 benchmark leader board. After 300

epochs of training however, CPL learns to down-sample the point cloud such that the

critical points of the object are mostly retained. In the context of point cloud

classification, by important points of an object we mean those points that contain the

necessary information to discriminate between different objects in the dataset. As seen,

the important points of each object for our classification task are still preserved even in

such small 64-point point clouds.

Claims

Adaptive Hierarchical Sampling for image Classification

The claims defining the invention are as follows:

In this patent, we propose a novel deterministic, adaptive, permutation-invariant

down-sampling layer, called Critical Points Layer (CPL), which learns to reduce the

number of points in an unordered point cloud while retaining the important (critical)

ones. Unlike most graph-based point cloud down-sampling methods that use k-NN to

find the neighboring points, CPL is a global down-sampling method, rendering it

computationally very efficient. The proposed layer can be used along with a graph

based point cloud convolution layer to form a convolutional neural network, dubbed

CP-Net in this patent.

The network in our present invention will be described further in detail.

(1) Critical Points Layer (CPL)

Let's assume the input to the CPL is an unordered point cloud with n points, each

represented as a feature vector xe Rd , where R is the set of real numbers and d is

the dimension of the feature vector. The goal of CPL is to generate a subset of input

points, called Critical Points (CP), with m: n points, each represented as a feature

vector ye R' , where 1 is the dimension of the new feature vector. The critical points

of a point cloud are the points with maximum information that are needed to be

preserved in a down-sampling (or pooling) process. These points may be changed based

on the task and application.

The block diagram of the proposed Critical Points Layer (CPL) is illustrated in

Figure la. The steps of the algorithm are explained in more details as follows:

Step 1: Get the maximumfeature value. The input point cloud Fs is a matrix with

n rows (corresponding to n input points) and d columns (corresponding to d

dimensional feature vectors). In the first step, the maximum feature value is obtained

for each column of the matrix Fs. This is the same as the max-pooling operation in

Point-Net. The resulting d-dimensional feature vector, denoted by ia., has the same

dimension as input feature vectors and can be independently used for classification and

segmentation tasks. However, we are interested in down-sampling the input points

rather than generating a single feature vector out of them. To this aim, the index of each

row with a maximum feature value is also saved in the index vector idx. Vector idx

contains the indices of all the points that have contributed to the feature vector fax

By definition, we call these points, the Critical Points (CP). These are the important

points that should be preserved in the down-sampling process.

Step 2: Set the unique indices. Index vector idx may contain multiple instances of

the same point. To avoid these repetitions, unique indices are extracted from idx, using

the "set (unique)" function. Output set which has the unique indices is called the Critical

Set (CS) and is denoted by uidx. Beside finding the unique vector, we also add up the

feature values from inax that correspond to the same point or index. Resulting feature

vector fs will be later used to sort the input points.

Step 3: Sort the feature vector fs . Next, feature vector fs is sorted (in an

ascending order). Corresponding indices in uidx are also rearranged based on the sorting

output, resulting in an index vector which is denoted by suidx. This step is necessary

for the following sampling (resizing) operation. It also makes CPL invariant to the order

of input points. Step 4: Resize and up-sample index vector suidx. Number of elements in suidx

may differ for different point clouds in the input batch. For batch processing however,

these numbers need to be the same. To address this, for each point cloud in the input

batch, the index vector suidx is up-sampled to a fixed size vector rsuidx using an up

sampling method for integer arrays, such as nearest neighbor resizing.

Step 5: Gatherpointsand correspondingfeature.As the final step, the up-sampled

index vector rsuidx, which contains the indices of all the critical points, is used to gather

points and their corresponding feature vectors. Since different feature vectors may

correspond to a single point, and because of the information being filtered in hidden

NN layers, we may want to gather the features from other layers (denoted by F,)than

those used for selecting the points (denoted by Fs ). However, critical points are defined based on the contribution of each point in the maximum feature vector obtained from Fs, thus here we use F, = Fs.

(2) Weighted Critical Points Layer (WCPL)

Step 1: Get the maximum feature value and set the unique indices. The maximum feature value is obtained for each column of the matrix F method the same as CPL's. To avoid these repetitions, On the basis of what CPL does, WCPL add a vector f, that records the number of contributions per point. Step 2: Sort thefeature vector fs and repeat. Next, feature vector fs is sorted and its corresponding indices in uidx are also rearranged based on the sorting output. To increase the weight of a point by a factor of C, we repeat the point index C times. By increasing the repetition frequency, the probability of selecting the point in the down-sampling process will also increase. From another point of view, in WCPL, the probability of missing a critical point in the output is lower than that in CPL. Then, for each point cloud in the input batch, the index vector midx is resized to afixed size vector rsuidx. Step 3: Gatherpoints and correspondingfeature.As the final step, the resized index vector rmidx, which contains the indices of all the critical points, is used to gather points and their corresponding feature vectors.

(3) Critical Points Net (CP-Net)

In this patent, we propose a hierarchical architecture to apply deep convolutional neural networks to point clouds, by systematically reducing the number of points using the proposed CPL/WCPL. In the proposed network model, named Critical Points Net (CP-Net), any graph convolution method can be used in convolution layers. The block diagram of CP-Net using EdgeCony as an example is shown in Figure 2.

Step 1: Pass the point cloud into a convolution layer. The input in Figure 2 is an unordered point cloud of size n. In the first step, the point cloud is passed into a convolution layer of choice to filter the input into a richer set of features. In the proposed network model, named Critical Points Net (CP-Net), any graph convolution method, such as DCNN, GCNN, MoNet or EdgeCony (from DGCNN) can be used in convolution layers. Step 2: Use the CPL/WCPL to down-sample the point cloud. The filtered output point cloud F, is then used as an input to the first CPL/WCPL. Using a CPL/WCPL with down-sampling factor ko , the number of points in Fs is reduced to n / k points. These steps are repeated for as many times as necessary to achieve a desired size of the point cloud (both in terms of number of points and feature vector size). Note that at j -th CPL/WCPL block, one can also benefit from using or concatenating features from all or some of the previous layers, i.e., {FI,F , --- ,F Y}, as long as they correspond to the same points. As a result, the number of output points will be n

Fig2. Fig1.

Fig3. 2021105154

Fig4.

Table 1.

Overall Mean Class Algorithm Accuracy (%) Accuracy (%) Vox-Net 83.00 85.9 ECC 83.2 - SO-Net 89.16 - Pointnet 89.20 86.0 Pointnet++ 90.70 - KCNet 91.0 - KD-Net 91.8 - DGCNN 91.84 89.40 RS-CNN 93.6 - Ours (CPNet) 92.33 89.90 Ours (WCPNet) 92.41 90.53