KR102388760B1 - Image convolution method in hyperbolic space - Google Patents

Abstract

The present invention relates to a method for performing an image convolution in consideration of a hierarchical relationship between hyperbolic feature vectors in a hyperbolic space. The image convolution method in the hyperbolic space according to one embodiment of the present invention comprises: a step of embedding an image feature vector in a Euclidean space into a hyperbolic feature vector in a hyperbolic space; a step of allocating a hierarchical weight for the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector; and a step of performing convolution of the hyperbolic feature vector by applying the hierarchical weight. Therefore, the present invention is capable of greatly improving a learning accuracy.

Description

IMAGE CONVOLUTION METHOD IN HYPERBOLIC SPACE

The present invention relates to a method for performing image convolution in consideration of the hierarchical relationship of hyperbolic feature vectors in a hyperbolic space.

Recently, in the field of deep learning using a convolutional neural network (CNN), a data learning method in a hyperbolic space has been highlighted because it can effectively form a hierarchical relationship.

In particular, as the need for deep learning for graph-structured data that can represent relationships between data emerges, data learning methods that consider hierarchical relationships have become a key research field to keep pace with the explosive growth of the AI field. is rising

However, the data learning method in the hyperbolic space studied so far has limitations in that it can be applied only to fields where data itself is designed to have a hierarchical structure, such as a knowledge graph or a synonym hierarchy. There was a problem of low usability.

An object of the present invention is to provide an image convolution method capable of assigning importance to each pixel constituting a hyperbolic feature vector in consideration of a hierarchical relationship between a hyperbolic feature vector embedded in a hyperbolic space.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned may be understood by the following description, and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the appended claims.

A convolution method in a hyperbolic space according to an embodiment of the present invention comprises the steps of embedding an image feature vector in Euclidean space into a hyperbolic feature vector in hyperbolic space, and the hyperbolic feature based on a hierarchical property of the hyperbolic feature vector. Allocating a hierarchical weight to a vector and applying the hierarchical weight to convolute the hyperbolic feature vector.

In an embodiment, the embedding comprises embedding each pixel constituting the image feature vector in the hyperbolic space through a mapping function.

In one embodiment, the embedding comprises embedding the image feature vector into a hyperbolic feature vector on the Poincare ball through a one-to-one correspondence function connecting the Euclidean space and a Poincare ball. characterized.

In an embodiment, the assigning the hierarchical weights comprises assigning the hierarchical weights based on a geodesic distance of the hyperbolic feature vector.

In an embodiment, the assigning the hierarchical weights comprises assigning the hierarchical weights inversely proportional to the geodesic distance of the hyperbolic feature vector.

In an embodiment, allocating the hierarchical weights may include allocating the hierarchical weights based on a distance between a reference pixel constituting the hyperbolic feature vector and the remaining pixels.

In an embodiment, allocating the hierarchical weights may include generating a hierarchical weight vector composed of the hierarchical weights.

In an embodiment, the allocating the hierarchical weights comprises: rearranging the hyperbolic feature vector according to the hierarchical weights; arranging the hierarchical weights according to the rearrangement order to generate a hierarchical weight vector; It is characterized in that it includes.

In an embodiment, the allocating the hierarchical weights includes: rearranging the hyperbolic feature vector in an order of magnitude of the hierarchical weights; generating a hierarchical weight vector arranged in an order of magnitude of the hierarchical weights; It is characterized in that it includes.

In an embodiment, the convolution includes multiplying a hierarchical weight vector composed of the hierarchical weights by the hyperbolic feature vector, and convolving the hyperbolic feature vector multiplied by the hierarchical weight vector. characterized in that

In the present invention, by assigning a hierarchical weight to each pixel constituting a hyperbolic feature vector, it is possible to give importance to a pixel having a high affinity in performing deep learning. Accordingly, it is possible not only to increase the learning effect and learning efficiency of the spatial propagation neural network, but also to significantly improve the learning accuracy for the runner parameters used in deep learning.

In addition, the present invention can rearrange the hyperbolic feature vector according to importance in assigning a weight to the hyperbolic feature vector. Accordingly, the arrangement of pixels in the hyperbolic feature vector may have a tendency, and when this is applied to a convolutional neural network, learning consistency and learning accuracy are improved, and a robust model can be built.

In addition to the above-described effects, the specific effects of the present invention will be described together while describing specific details for carrying out the invention below.

1 is a flowchart schematically illustrating an image convolution method of the present invention;
2 is a diagram illustrating a state in which two vectors in a Euclidean space are embedded in a hyperbolic space.
3 is a diagram illustrating a state in which two vectors on a Euclidean space are embedded in a Poincare ball.
4 is a diagram for explaining a geodesic distance in a hyperbolic space.
5 is a flowchart illustrating an image convolution method in a hyperbolic space according to an embodiment of the present invention.
6 is a flowchart illustrating an image convolution method in a hyperbolic space according to another embodiment of the present invention.
7 is a diagram for comparing the image reproduction accuracy of the present invention and the prior art.

The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

In the present specification, “upper (or lower)” or “above (or lower)” of a component means that any component is disposed in contact with the upper surface (or lower surface) of the component, as well as that any component is disposed in contact with the component. , may mean that another component may be interposed between the component and any component disposed on (or under) the component.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps.

In addition, in this specification, when "A and / or B" is said, unless otherwise stated, it means A, B or A and B, and when said "C to D", it is Unless otherwise specified, it means C or more and D or less.

The present invention relates to a method of performing image convolution in consideration of a hierarchical relationship between hyperbolic feature vectors in a hyperbolic space. More specifically, the present invention is a method applied to a neural network using spatial propagation, and may be used to spatially propagate a sparse input to generate a dense output.

Hereinafter, the image convolution method of the present invention will be described in detail with reference to FIGS. 1 to 7 .

1 is a flowchart schematically illustrating an image convolution method of the present invention.

2 is a diagram illustrating a state in which two vectors in the Euclidean space are embedded in a hyperbolic space, and FIG. 3 is a diagram illustrating a state in which two vectors in the Euclidean space are embedded in a Poincare ball.

4 is a diagram for explaining a geodesic distance in a hyperbolic space.

5 is a flowchart illustrating an image convolution method in a hyperbolic space according to an embodiment of the present invention, and FIG. 6 is a flowchart illustrating an image convolution method in a hyperbolic space according to another embodiment of the present invention.

7 is a diagram for comparing the image reproduction accuracy of the present invention and the prior art.

Referring to FIG. 1 , an image convolution method in a hyperbolic space, which will be described later, converts an image feature vector on an Euclidean space to a hyperbolic feature vector on a hyperbolic space. ) may include the step of embedding (S110).

In addition, the image convolution method may include allocating a hierarchical weight to the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector ( S120 ). In addition, the image convolution method may include the step of convolving the hyperbolic feature vector by applying the previously assigned hierarchical weight ( S130 ).

Each of the steps S110 to S130 described above may be programmed and implemented through software, or may be implemented through hardware including a processing unit such as a GPU (Graphic Processing Unit) and CPU (Central Processing Unit). Hereinafter, for convenience of description, it is assumed that each step (S110 to S130) of the invention is performed by a convolution module.

Hereinafter, each step shown in FIG. 1 will be described in detail.

The convolution module may receive the image feature vector in the Euclidean space and embed it in the hyperbolic space (S110). More specifically, the convolution module may convert an image feature vector located on the Euclidean space into a hyperbolic feature vector located on the hyperbolic space.

The hyperbolic space is a non-euclidean space, and may be a homogeneous space having a uniform negative curvature at all points in the space. A hyperbolic space may contain multidimensional vectors that are difficult to shape.

In the following drawings, it is assumed that the hyperbolic space is expressed in three dimensions in order to shape the hyperbolic space. However, as described above, the hyperbolic space may include a multidimensional vector of four or more dimensions.

Referring to FIG. 2 , the convolution module can embed two image feature vectors (F ₁ , F ₂ ) located in a two-dimensional Euclidean space into two hyperbolic feature vectors (H ₁ , H ₂ ) located in a hyperbolic space. there is. To this end, the convolution module may use a mapping function (M).

The mapping function M may be a one-to-one correspondence function for mapping the Euclidean space to the hyperbolic space or for mapping the hyperbolic space to the Euclidean space. For example, the mapping function M may be defined as a linear function, and in this case, each coefficient constituting the linear function may be a learnable parameter.

ball)으로 모델링될 수 있다. 이 때, 컨볼루션 모듈은 유클리디언 공간과 푸앵카레 공을 연결하는 매핑함수(M), 구체적으로 일대일 대응함수를 통해 두 이미지 특징 벡터(F₁, F₂)를 푸앵카레 공 상의 쌍곡 특징 벡터(H₁, H₂)에 임베딩할 수 있다. 여기에서도 매핑함수(M)는 선형 함수로 정의될 수 있고, 이 때 선형 함수를 구성하는 각 계수는 러너블 파라미터일 수 있다.Meanwhile, referring to FIG. 3 , the aforementioned hyperbolic space is a Poincaré ball (Poincar

ball) can be modeled. At this time, the convolution module converts the two image feature vectors (F ₁ , F ₂ ) to the hyperbolic feature vector (H ₁ , H ₂ ) can be embedded. Here again, the mapping function M may be defined as a linear function, and in this case, each coefficient constituting the linear function may be a runnerable parameter.

) 합 연산 및 곱 연산이 이용될 수 있다. 보다 구체적으로, 뫼비우스 곱 연산은 하기 [수학식 1]과 같이 표현될 수 있고, 뫼비우스 합 연산은 하기 [수학식 2]와 같이 표현될 수 있다.On the other hand, since the multiplication and sum operations constituting the mapping function (M) are performed in hyperbolic space, the Möbius (

) sum operation and multiplication operation may be used. More specifically, the Möbius product operation may be expressed as in [Equation 1], and the Möbius sum operation may be expressed as [Equation 2] below.

는 유클리디언 놈(norm))(M is the matrix, u is the vector, k is the curvature of the Poincare ball,

is the Euclidean norm)

는 유클리디언 내적(inner product))(u, v are vectors, k is the curvature of the Poincare ball,

is the Euclidean inner product)

As described above, when embedding is completed, the convolution module may allocate a hierarchical weight to the hyperbolic feature vector based on the hierarchical property of the hyperbolic feature vector (S120).

Here, the hierarchical property is a property related to similarity and/or affinity between hyperbolic feature vectors. More specifically, the hierarchical attribute may include any parameter indicating an arbitrary value or degree regarding the similarity and/or affinity of vectors in the hyperbolic space.

The mapping function M described in step S110 assumes that the hyperbolic feature vectors are uniformly distributed in the hyperbolic space just as the image feature vectors are arranged on a regular grid in the Euclidean space. However, since the hyperbolic feature vector is projected into a manifold space through dimensionality reduction for data visualization, the above premise may not be satisfied. Accordingly, when a hyperbolic feature vector is convolved without additional data processing, there is a problem in that non-Euclidean spatial characteristics cannot be faithfully reflected.

To alleviate this problem, the convolution module may assign a hierarchical weight to each hyperbolic feature vector based on the geodesic distance of the hyperbolic feature vector.

Referring to FIG. 4 , the geodesic distance in the hyperbolic space may be different from the Euclidean distance in the Euclidean space. Euclidean distance means a straight line distance between two pixels (H(x ₁ , y ₁ ), H(x ₂ , y ₂ )), whereas geodesic distance can mean a curved distance through the surface of a hyperbolic surface. .

The convolution module may calculate a hyperbolic feature vector embedded in a hyperbolic space, specifically, a geodesic distance between pixels constituting the hyperbolic feature vector, and may assign a hierarchical weight according to the calculated distance.

)로 임베딩될 수 있다. 이에 따라, 인접 픽셀과의 거리가 정규화되어 있던 이미지 특징 벡터는, 인접 픽셀과의 거리가 각기 다른 쌍곡 특징 벡터(

)로 매핑될 수 있다.Taking Fig. 5 as an example, the image feature vector on the Euclidean space is converted to the hyperbolic feature vector on the Poincar ball by the mapping function (M).

) can be embedded. Accordingly, the image feature vector in which the distance to the adjacent pixel is normalized is a hyperbolic feature vector (

) can be mapped to

) 내의 각 픽셀간의 지오데식 거리를 산출하고, 산출된 거리에 기초하여 각 픽셀에 계층적 가중치를 할당할 수 있다.The convolution module is a hyperbolic feature vector (

) may be calculated, and a hierarchical weight may be assigned to each pixel based on the calculated distance.

The geodesic distance may be calculated according to Equation 3 below.

는 유클리디언 놈)(d _k is the geodesic distance, u and v are two points on the Poincare ball, k is the curvature of the Poincare ball,

is a Euclidian

The closer the geodesic distance between hyperbolic feature vectors in the hyperbolic space is, the higher the affinity between each pixel may be. In a neural network using spatial propagation, neural network learning can be performed in consideration of such affinity.

) 간의 지오데식 거리에 반비례하는 계층적 가중치를 할당할 수 있다. 이 때, 계층적 가중치는 지오데식 거리에 반비례하도록 정규화(normalize)될 수 있다.The convolution module of the present invention may give higher importance to pixels having high affinity in order to improve the learning efficiency of a spatial propagation neural network (SPN). To this end, the convolution module uses a hyperbolic feature vector (

) can be assigned a hierarchical weight that is inversely proportional to the geodesic distance between them. In this case, the hierarchical weight may be normalized to be inversely proportional to the geodesic distance.

)에 계층적 가중치를 할당하는 단계(S120)를 구체적으로 설명하도록 한다.Hereinafter, with reference to FIG. 5, a hyperbolic feature vector (

) of allocating hierarchical weights ( S120 ) will be described in detail.

)를 구성하는 기준 픽셀(rf)와 나머지 픽셀(nf)간의 거리에 기초하여 계층적 가중치를 할당할 수 있다. 예컨대, 컨볼루션 모듈은 도 5에 도시된 붉은색 픽셀을 기준 픽셀(rf)로 설정할 수 있고, 기준 픽셀(rf)와 나머지 픽셀(nf)간의 거리를 산출할 수 있다.The convolution module is a hyperbolic feature vector (

), a hierarchical weight may be assigned based on the distance between the reference pixel rf and the remaining pixels nf. For example, the convolution module may set the red pixel shown in FIG. 5 as the reference pixel rf, and may calculate a distance between the reference pixel rf and the remaining pixels nf.

Referring to the position of the pixel embedded in the Poincare ball, the geodesic distance between the red reference pixel (rf) and the green pixel (nf1) may be the shortest, and the geodesic distance between the red reference pixel (rf) and the orange pixel (nf2) may be the shortest. The desicing distance may be the longest.

Since the red reference pixel (rf) has a geodesic distance of 0, the convolution module can set the hierarchical weight for the red reference pixel (rf) to the maximum value, and the distance from the green pixel (nf1) Since is closer than the distance to the orange pixel nf2, the hierarchical weight of the green pixel nf1 may be set higher than the hierarchical weight of the orange pixel nf2.

)를 생성할 수 있다.The convolution module constructs a hierarchical weight vector (

) can be created.

)를 생성하는 방법에 대해 설명하도록 한다. 도 5에서는 설명의 편의를 위해 계층적 가중치의 크기를, 계층적 가중치의 크기에 비례하여 짙어지는 음영으로 표현하였다.Referring back to Fig. 5, the hierarchical weight vector (

) to explain how to create it. In FIG. 5, for convenience of explanation, the size of the hierarchical weight is expressed as a shade that becomes darker in proportion to the size of the hierarchical weight.

)와 크기(width, height)가 동일한 계층적 가중치 벡터(

)를 생성할 수 있다. 예컨대, 도 5에 도시된 바와 같이 쌍곡 특징 벡터(

)가 3 x 3 매트릭스 형태일 때, 계층적 가중치 벡터(

) 역시 3 x 3 매트릭스 형태를 가질 수 있다. 이 때, 계층적 가중치 벡터(

)는 쌍곡 특징 벡터(

) 내 각 픽셀에 대한 계층적 가중치를 나타내는 픽셀 값으로 구성될 수 있다.The convolution module is a hyperbolic feature vector (

) and a hierarchical weight vector (

) can be created. For example, as shown in Fig. 5, the hyperbolic feature vector (

) is in the form of a 3 x 3 matrix, the hierarchical weight vector (

) may also have a 3 x 3 matrix form. In this case, the hierarchical weight vector (

) is the hyperbolic feature vector (

) may consist of a pixel value representing a hierarchical weight for each pixel in

)의 2행 2열의 픽셀 값이 가장 높게 설정될 수 있다. 또한, 초록색 픽셀(nf1)(3행 2열)에 대한 계층적 가중치가 다음으로 높으므로 계층적 가중치 벡터(

)의 3행 2열의 픽셀 값이 다음으로 높게 설정될 수 있다. 한편, 오렌지색 픽셀(nf2)(1행 3열)에 대한 계층적 가중치가 최소이므로 계층적 가중치 벡터(

)의 1행 3열의 픽셀 값이 가장 낮게 설정될 수 있다.As described above, since the hierarchical weight for the red reference pixel rf (2 rows, 2 columns) is the maximum, in FIG.

) may be set to the highest pixel value in row 2, column 2, and so on. Also, since the hierarchical weight for the green pixel (nf1) (3 rows, 2 columns) is the next highest,

), the pixel value in row 3, column 2 may be set to the next highest value. On the other hand, since the hierarchical weight for the orange pixel (nf2) (1 row 3 column) is minimal,

), a pixel value in row 1, column 3, and column 1 may be set to the lowest value.

)에 대응하는 계층적 가중치 벡터(

)를 생성할 수 있다.As described above, the convolution module uses the hierarchical weights assigned based on the geodesic distance to calculate each hyperbolic feature vector (

) corresponding to the hierarchical weight vector (

) can be created.

)에 계층적 가중치를 할당함으로써 지오데식 거리가 가까운, 다시 말해, 어피니티가 높은 픽셀의 중요도를 더 높게 설정할 수 있다. 이에 따라, 본 발명에 의하면 공간 전파 신경망의 학습 효과 및 학습 효율성을 증대시킬 수 있을 뿐만 아니라 컨볼루션에 이용되는 필터 및/또는 커널(kernel)내 러너블 파라미터에 대한 학습 정확도 또한 크게 향상시킬 수 있다.The present invention is a hyperbolic feature vector (

) by assigning a hierarchical weight, it is possible to set the importance of pixels having a close geodesic distance, that is, having a high affinity, to be higher. Accordingly, according to the present invention, it is possible to not only increase the learning effect and learning efficiency of the spatial propagation neural network, but also greatly improve the learning accuracy of the filter used for convolution and/or the runner parameter in the kernel. .

) 내 각 픽셀 값들이 쌍곡 특징 벡터(

)의 계층적 속성에 따라 무작위로(randomly) 배치될 수 있다. 구체적으로, 도 5에 도시된 것과는 달리 만약 붉은색 기준 픽셀(rf)와 초록색 픽셀(nf1)간의 지오데식 거리가 가장 멀다고 가정하면, 계층적 가중치 벡터(

)의 3행 2열의 픽셀 값이 가장 낮게 설정될 수 있다. 이와 같은 계층적 가중치 벡터(

)의 무작위성(randomness)에 의해 딥러닝의 학습 일관성이 다소 떨어질 수 있다는 한계가 있다.Meanwhile, according to the hierarchical weight allocation method described with reference to FIG. 5, the hierarchical weight vector (

) is a hyperbolic feature vector (

) may be randomly arranged according to the hierarchical property of . Specifically, unlike shown in FIG. 5, if it is assumed that the geodesic distance between the red reference pixel rf and the green pixel nf1 is the longest, the hierarchical weight vector (

), the pixel value of 3 rows and 2 columns may be set to the lowest value. A hierarchical weight vector like this (

), there is a limit that the learning consistency of deep learning may be somewhat lowered due to the randomness of it.

)를 재배치할 수 있고(S121), 재배치 순서에 따라 계층적 가중치를 배열하여 계층적 가중치 벡터(

)를 생성할 수 있다(S122).In order to solve the above-mentioned potential problems, the convolution module uses the hyperbolic feature vector (

) can be rearranged (S121), and the hierarchical weight vector (

) can be generated (S122).

)를 생성하는 또 다른 방법에 대해 설명하도록 한다. 도 6에서도 설명의 편의를 위해 계층적 가중치의 크기를, 계층적 가중치의 크기에 비례하여 짙어지는 음영으로 표현하였다.Hereinafter, with reference to FIG. 6, a hierarchical weight vector (

), another way to create Also in FIG. 6, for convenience of explanation, the size of the hierarchical weight is expressed as a shade that becomes darker in proportion to the size of the hierarchical weight.

)를 구성하는 픽셀간의 계층적 속성, 예컨대 지오데식 거리에 기초하여 각 픽셀에 대한 계층적 가중치를 결정할 수 있다.As described with reference to FIG. 5, the convolution module is a hyperbolic feature vector (

), a hierarchical weight for each pixel may be determined based on a hierarchical property between pixels constituting the pixel, for example, a geodesic distance.

)를 재배치할 수 있다. 보다 구체적으로, 컨볼루션 모듈은 계층적 가중치가 큰 순서대로 또는 작은 순서대로 쌍곡 특징 벡터(

) 내 픽셀의 위치를 재배치할 수 있다.Then, the convolution module constructs a hyperbolic feature vector (

) can be rearranged. More specifically, the convolution module calculates hyperbolic feature vectors (

) to reposition my pixels.

Referring to FIG. 6 , based on the geodesic distance on the Poincare ball, the weight for the red reference pixel rf may be set the largest, and the weight for the green pixel nf1 may be set to the second largest. . Meanwhile, the weight of the orange pixel nf2 may be set to the smallest value.

)의 각 픽셀을 계층적 가중치가 큰 순서대로 1행 1열부터 1행 3열, 2행 1열부터 2행 3열, 3행 1열부터 3행 3열에 배치할 수 있다. 이에 따라, 도 6에 도시된 바와 같이 계층적 가중치가 가장 큰 붉은색 기준 픽셀(rf)는 1행 1열에 배치될 수 있고, 계층적 가중치가 그 다음으로 큰 초록색 픽셀(nf1)는 1행 2열에 배치될 수 있으며, 계층적 가중치가 가장 작은 오렌지색 픽셀(nf2)는 3행 3열에 배치될 수 있다.At this time, the convolution module is a hyperbolic feature vector (

) may be arranged in the order of increasing the hierarchical weight in the order of increasing the hierarchical weight: 1 row 1 column 1 row 3 columns, 2 rows 1 column 2 rows 3 columns, 3 rows 1 column 3 rows 3 columns. Accordingly, as shown in FIG. 6 , the red reference pixel rf having the largest hierarchical weight may be arranged in row 1, column 1, and the green pixel nf1 having the next largest hierarchical weight is row 1, 2 It may be arranged in a column, and the orange pixel nf2 having the smallest hierarchical weight may be arranged in 3 rows and 3 columns.

Unlike the above, the convolution module can rearrange each pixel in the order of the smallest hierarchical weight, and from 1 row 1 column to 3 row 1 column, 1 row 2 column to 3 row 2 column, 1 row 3 column to 3 Pixels may be arranged in the order of rows and columns.

)를 생성할 수 있다. 다시 말해, 컨볼루션 모듈은 계층적 가중치 벡터(

)를 생성하되, 내부 각 픽셀 값(계층적 가중치)이 크기 순서대로 배열되도록 할 수 있다. 이 때, 계층적 가중치 벡터(

) 내 픽셀의 배열 방향은 전술한 쌍곡 특징 벡터(

)의 재배치 방향과 동일할 수 있다.Then, the convolution module constructs a hierarchical weight vector (

) can be created. In other words, the convolution module is a hierarchical weight vector (

), but with each inner pixel value (hierarchical weight) arranged in size order. In this case, the hierarchical weight vector (

) the arrangement direction of the pixels in the above-mentioned hyperbolic feature vector (

) can be the same as the rearrangement direction.

)의 각 픽셀을 그 값(계층적 가중치)이 큰 순서에 따라 배열할 수 있다. 이에 따라, 붉은색 기준 픽셀(rf)에 대응하는 계층적 가중치는 1행 1열에 배열될 수 있고, 초록색 픽셀(nf1)에 대응하는 계층적 가중치는 1행 2열에 배치될 수 있으며, 오렌지색 픽셀(nf2)에 대응하는 계층적 가중치는 3행 3열에 배치될 수 있다.Referring back to Figure 6, the convolution module is a hierarchical weight vector (

) can be arranged according to the order of the largest value (hierarchical weight). Accordingly, the hierarchical weights corresponding to the red reference pixel rf may be arranged in one row and one column, the hierarchical weights corresponding to the green pixel nf1 may be arranged in the first row and second column, and the orange pixel ( Hierarchical weights corresponding to nf2) may be arranged in 3 rows and 3 columns.

) 생성이 완료되면 컨볼루션 모듈은 계층적 가중치를 적용하여 쌍곡 특징 벡터(

)를 컨볼루션할 수 있다(S130). 보다 구체적으로, 컨볼루션 모듈은 먼저 쌍곡 특징 벡터(

)에 계층적 가중치를 적용한 후, 계층적 가중치가 적용된 쌍곡 특징 벡터(

)를 컨볼루션할 수 있다.Hierarchical weight assignment and/or hierarchical weight vector (

), the convolution module applies hierarchical weights to create hyperbolic feature vectors (

) can be convolved (S130). More specifically, the convolution module first constructs a hyperbolic feature vector (

) is hierarchically weighted, and then the hierarchically weighted hyperbolic feature vector (

) can be convolved.

)가 생성된 경우, 컨볼루션 모듈은 계층적 가중치 벡터(

)와 쌍곡 특징 벡터(

)를 곱함으로써 쌍곡 특징 벡터(

)에 계층적 가중치를 적용할 수 있다. 여기서 곱 연산에는 전술한 [수학식 1]로 표현되는 뫼비우스 곱 연산이 이용될 수 있다.5 and 6, as described above, the hierarchical weight vector (

) is generated, the convolution module

) and the hyperbolic feature vector (

) by multiplying the hyperbolic feature vector (

) can be applied with hierarchical weights. Here, the Möbius product operation expressed by the above-mentioned [Equation 1] may be used for the multiplication operation.

)를 컨볼루션할 수 있다. 이 때, 컨볼루션에는 해당 기술분야에서 이용되는 다양한 기법들이 적용될 수 있으며, 특히 공간 전파 신경망(SPN)에서 이용되는 기법이 이용될 수 있다.Then, the convolution module constructs a hierarchically weighted hyperbolic feature vector (

) can be convolved. In this case, various techniques used in the relevant technical field may be applied to convolution, and in particular, a technique used in a spatial propagation neural network (SPN) may be used.

)와 계층적 가중치 벡터(

) 내 각 픽셀 값들은 크기에 따라 규칙적으로(regularly) 배치될 수 있다. 이에 따라, 쌍곡 특징 벡터(

)에 계층적 가중치 벡터(

)를 곱한 출력 역시 그 픽셀이 중요도 순으로 배치될 수 있다.According to the hierarchical weighting method described with reference to FIG. 6, the hyperbolic feature vector (

) and the hierarchical weight vector (

) in each pixel value may be arranged regularly according to the size. Accordingly, the hyperbolic feature vector (

) in the hierarchical weight vector(

) multiplied by the pixel may also be arranged in order of importance.

The arrangement of pixels in the output may have a tendency according to importance, and when this is applied to a convolutional neural network, learning consistency and learning accuracy are improved, so that a robust model can be built.

Referring to FIG. 7 , according to the present invention, dense depth information may be generated using sparse depth information through a spatial propagation neural network (SPN). To this end, the present invention may receive a depth sample as an image feature vector and embed it in a hyperbolic space.

)를 컨볼루션할 수 있다.Then, the present invention can apply a weight to each pixel based on the geodesic distance between pixels in the embedded depth sample, and apply a weighted hyperbolic feature vector (

) can be convolved.

In light of the convolution result of the present invention shown in FIG. 7, a remarkable research achievement in the field of recent spatial propagation neural networks (SPN), 『Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolutional spatial propagation network. When the present invention is compared with the Convolutional Spatial Propagation Network (CSPN) introduced in In Proceedings of European Conference on Computer Vision (ECCV), 2018., restoration using the convolutional method of the present invention It can be seen that the image obtained is more similar to GT (Ground Truth).

As described above, the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed in the present specification. It is obvious that variations can be made. In addition, although the effects according to the configuration of the present invention have not been explicitly described and described while describing the embodiments of the present invention, it is natural that the effects predictable by the configuration should also be recognized.

Claims (10)

In the image convolution method performed by the convolution module,
embedding an image feature vector in Euclidean space into a hyperbolic feature vector in hyperbolic space;
setting a hierarchical weight for the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector;
rearranging the hyperbolic feature vectors in an order of magnitude of the hierarchical weights;
generating a hierarchical weight vector in which the hierarchical weights are arranged in order of magnitude; and
Convolving the hyperbolic feature vector after applying the hierarchical weight vector to the hyperbolic feature vector
Image convolution method in hyperbolic space.
According to claim 1,
The embedding step
and embedding each pixel constituting the image feature vector in the hyperbolic space through a mapping function.
Image convolution method in hyperbolic space.
According to claim 1,
The embedding step
The Euclidean space and the Poincaré ball

and embedding the image feature vector into a hyperbolic feature vector on the Poincare ball through a one-to-one correspondence function connecting balls)
Image convolution method in hyperbolic space.
According to claim 1,
The step of setting the hierarchical weight is
setting a hierarchical weight based on the geodesic distance of the hyperbolic feature vector
Image convolution method in hyperbolic space.
According to claim 1,
The step of setting the hierarchical weight is
Setting a hierarchical weight inversely proportional to the geodesic distance of the hyperbolic feature vector
Image convolution method in hyperbolic space.
According to claim 1,
The step of setting the hierarchical weight is
setting a hierarchical weight based on a distance between a reference pixel constituting the hyperbolic feature vector and the remaining pixels
Image convolution method in hyperbolic space.
delete delete delete According to claim 1,
The convolution step is
multiplying the hierarchical weight vector composed of the hierarchical weights and the hyperbolic feature vector;
Convolving a hyperbolic feature vector multiplied by the hierarchical weight vector
Image convolution method in hyperbolic space.

Images