KR20210096447A - The method, apparatus and the program for image region segmentation - Google Patents

Abstract

The present invention provides a method, apparatus, and program for segmenting an image region that can perform image region segmentation faster and more accurately. The method includes the steps of: receiving image data as input by a processor of the computer; converting, by the processor, the image data into a latent space phenotype; converting, by the processor, the data converted into the latent spatial phenotype into a spike train with respect to time; performing learning by inputting, by the processor, the data converted into the spike sequence into an SNN-based learning module; extracting, by the processor, a spike sequence with respect to time with respect to the output data of the learning module; converting, by the processor, the extracted spike sequence into a latent space phenotype; and outputting, by the processor, the data converted into the latent space phenotype as region-segmented image data.

Description

Image region segmentation method, image region segmentation apparatus, and image region segmentation program {The method, apparatus and the program for image region segmentation}

The present invention relates to a method, apparatus, and program for segmenting regions in image data.

In the field of image-related technology, a technology for recognizing an object such as a human face using an image has recently been developed. In order to recognize an object such as a face, it is necessary to extract a portion excluding the background from the image.

In order to recognize an object from an image, it is necessary to segment a region within the image, and label which object (eg, class) the image corresponds to using the segmented region.

At this time, a semantic segmentation supervised learning method is used when segmenting a region within an image, and this semantic segmentation supervised learning has many problems at the pixel level.

As a specific problem, many pixels of an object correspond to a single class, and a single input image may have multiple classes spread across pixels.

(KR) Patent Publication No. 10-2019-0112378 (KR) Patent Publication No. 10-2016-0138042 (KR) Patent Publication No. 10-2017-0038622

The present invention has been devised to solve the above problem, and a Semantic segmentation supervised learning algorithm based on a spiking neural network (SNN) of a supervised learning method in various datasets. We propose a method for segmenting an image region, an apparatus for segmenting an image region, and a program for segmenting an image region in which this algorithm is reflected.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

An image region segmentation method according to an embodiment of the present invention includes: receiving, by a computer processor, image data as an input; converting, by the processor, the image data into a latent space phenotype; converting, by the processor, the data converted into the latent spatial phenotype into a time spike train; performing learning by the processor by inputting the data converted into the spike sequence into an SNN-based learning module; extracting, by the processor, a spike sequence with respect to time with respect to the output data of the learning module; converting, by the processor, the extracted spike sequence into a latent space phenotype; and outputting, by the processor, the data converted into the latent spatial phenotype as region-divided image data.

An image region segmentation method according to an embodiment of the present invention uses an encoder block in the step of the processor converting the image data into a latent space phenotype, and the processor uses the extracted spike sequence It may be to use a decoder block in the step of converting to a latent space phenotype.

In the image region segmentation method according to an embodiment of the present invention, the ground-truth image is also converted into a latent space phenotype in the step of the processor performing learning, and the converted data is spiked with respect to time It may be characterized by using input data transformed into a spike train.

In the image region segmentation method according to an embodiment of the present invention, in the step of the processor performing learning, a weight using a loss error between an input spike synapse and an output spike synapse is applied and updated.

The present invention may include an image region dividing apparatus and an image region dividing program for performing the above-described image region dividing method.

When the image region segmentation method to which the algorithm is applied according to an embodiment of the present invention is used, there is an advantage that image region segmentation can be performed faster and more accurately than the conventional method.

Effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a flowchart of a method for segmenting an image region according to an embodiment of the present invention.
2 shows a process of converting input image data into a latent space phenotype according to an embodiment of the present invention.
3 shows a process of converting a spike sequence with respect to time according to an embodiment of the present invention.
4 shows a process of converting a spike sequence with respect to time according to an embodiment of the present invention.
5 shows a process of performing learning in an SNN-based learning module according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments will be described in detail with reference to the drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. and/or includes a combination of a plurality of related description items or any of a plurality of related description items.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that other components may exist in between. something to do. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

It should also be understood that the terms "first" and "second" are used herein for distinguishing purposes only, and are not meant to indicate or anticipate sequences or priorities in any way.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Throughout the specification and claims, when a part includes a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart of a method for segmenting an image region according to an embodiment of the present invention.

Referring to FIG. 1 , a method for segmenting an image region according to an embodiment of the present invention includes: receiving, by a computer processor, image data as an input (S100); converting, by the processor, the image data into a latent space phenotype (S110); converting, by the processor, the data converted into the latent spatial phenotype into a time spike train (S120); performing, by the processor, learning by inputting the data converted into the spike sequence into an SNN-based learning module (S130); extracting, by the processor, a spike sequence with respect to time with respect to the output data of the learning module (S140); converting, by the processor, the extracted spike sequence into a latent space phenotype (S150); and outputting, by the processor, the data converted into the latent spatial phenotype as region-divided image data (S160).

2 shows a process of converting input image data into a latent space phenotype according to an embodiment of the present invention.

Transforming the image data into a latent spatial phenotype may be performed using an encoder block.

The latent space according to an embodiment of the present invention means reducing the dimension of real data in order to extract features necessary for machine learning, and can explain the analysis target well.

The upper diagram of FIG. 2 shows conversion of input RGB data to be analyzed into a latent space phenotype 20 using an encoder. The figure at the bottom of FIG. 2 shows the transformation of ground-truth data into a latent space phenotype 22 .

3 shows a process of converting into a spike sequence with respect to time according to an embodiment of the present invention, and FIG. 4 shows a detailed process of converting it into a spike sequence.

Referring to FIG. 3 , the data 20 and 22 converted to the latent space phenotype are converted into spike columns 30 and 32 for time, and both the input data to be analyzed and the ground-truth data are converted into spike columns. After the transformation, it is input to the SNN (Spiking Neural Network)-based learning module 100 .

5 shows a process of performing learning in the SNN-based learning module 100 according to an embodiment of the present invention.

Referring to FIG. 5 , the network architecture consists of a spike response model network of spiking neurons having connections implemented with a plurality of delayed synaptic terminals 50 .

Here, the input data 30 to be analyzed converted into a spike column can be derived as output resultant spikes 34 through SNN learning, and ground-truth data converted into a spike column is the target output spike. (desired output spikes, GT) 32 .

When the firing time of neuron h is t _h , the presynaptic neuron set can be set as follows.

는 시냅스 가중치 w _ho 에 의해 가중되지 않은 지연

을 갖는 반응형 스파이크 함수

에 의해 아래 식 1와 같이 결정될 수 있다.Input synaptic current from neuron h to neuron o

is the delay unweighted by the synaptic weight w _ho

Responsive spike function with

can be determined as in Equation 1 below.

[Equation 1]

는 시냅스 가중치 w _ho 에 의해 가중되는 반응형 스파이크 함수

에 의해 아래 식 2와 같이 결정될 수 있다. Output synaptic current from neuron h to neuron o

is a reactive spike function weighted by the synaptic weight w _ho

can be determined as in Equation 2 below.

[Equation 2]

In the multi-delayed synaptic terminal 50, Equation 2 provides all connections between the neuron h and the neuron o for all n terminals, so it can be determined as in Equation 3 below.

[Equation 3]

)을 초과 할 때 뉴런 스파이크로서 지수 붕괴 함수를 사용하여 시냅스 후 스파이크의 응답성을 모델링 할 수 있다. 따라서, 스파이크 함수

는 식 4와 같이 표현될 수 있다.As the learning model was developed using the spike response model, the synaptic current

), we can use the exponential decay function as the neuron spike to model the responsiveness of the post-synaptic spike. So, the spike function

can be expressed as Equation 4.

[Equation 4]

는 막 전위의 붕괴 시간 상수이고, 막 전위의 임계 값 (

)은 모든 뉴런에 대해 동일하다.here

is the decay time constant of the membrane potential, the threshold of the membrane potential (

) is the same for all neurons.

Since the learning model contains two types of connections: single synapses and multi-synapses, weight transfer is performed appropriately for single synapses and multiple synapses.

The learning process according to an embodiment of the present invention may accurately calculate an error according to an utterance time.

}에서 목적 출력 스파이크(32, desired output spikes) {

}를 뺀 것으로 정의될 수 있다.The error is output resultant spikes (34, output resultant spikes) {

} in the desired output spikes(32, desired output spikes) {

} can be defined by subtracting .

} - Desired output spikes {

} Error = Output resultant spikes {

} - Desired output spikes {

}

On the other hand, the spike column 30 converted into the latent spatial phenotype can be derived as output resultant spikes 34 through the learning model, and the desired output spikes 32 are measured as the spike column converted It means ground-truth data.

The error objective function may be one of mainly available loss functions, and the error objective function can be expressed as Equation 5 below.

[Equation 5]

하여 손실 에러(ε)를 업데이트 할 수 있다.In the case of backpropagation, the inter-synaptic weights must be adjusted to yield a minimum error between the output result spike 34 and the target output spike 32 . Therefore, using Equation 7 which is different from Equation 6 below

to update the loss error (ε).

[Equation 6]

[Equation 7]

는 학습률이며, ( t _o )는 출력의 발사시간 ( O _o )는 출력 시냅스 전류이고, 식 7에서 점선으로 강조 표시된 용어를 연쇄 규칙(Chain-rule)을 활용하는 데 사용하면 아래 식 8 및 식 9와 같이 산출될 수 있다.

is the learning rate, ( t _o ) is the firing time of the output ( O _o ) is the output synaptic current, and the terms highlighted by the dashed line in Equation 7 are used to utilize the Chain-rule, Equation 8 and Equation 8 below 9 can be calculated.

[Equation 8]

[Equation 9]

If Equation 8 and Equation 9 are combined, Equation 6 can be expressed as Equation 10 below.

[Equation 10]

If the above Equation 10 is briefly expressed, it can be expressed as Equation 11 below.

[Equation 11]

Here, β _o can be expressed as Equation 12 below.

[Equation 12]

Equations 11 and 12 can be used for back propagation corresponding to weighted adaptation to neurons of the output layer.

Similarly, we can perform error backpropagation (weight adjustment) for hidden layer neurons.

According to an embodiment of the present invention, weights can be updated using error backpropagation regardless of the number of hidden layers, and when weights are updated in a series of training samples, a fixed weight can be used in a test scenario.

The above description is merely illustrative of the technical idea of the present invention, and a person of ordinary skill in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

20: latent space phenotype input data
22: Latent space phenotypic data for ground-truth data
30: input data converted to spike column
32: Actual data converted to spike column
34: output result spike
50: multiple delayed synaptic terminals
100: SNN-based learning module

Claims (6)

In the image region segmentation method,
receiving image data as input by a processor of the computer;
converting, by the processor, the image data into a latent space phenotype;
converting, by the processor, the data converted into the latent spatial phenotype into a time spike train;
performing, by the processor, learning by inputting the data converted into the spike sequence into an SNN-based learning module;
extracting, by the processor, a spike sequence with respect to time with respect to the output data of the learning module;
converting, by the processor, the extracted spike sequence into a latent space phenotype; and
and outputting, by the processor, the data converted into the latent spatial phenotype as region-segmented image data. According to claim 1,
Using an encoder block in the step of the processor converting the image data into a latent space phenotype,
The image region segmentation method of claim 1, wherein the processor uses a decoder block in the step of converting the extracted spike sequence into a latent space phenotype. According to claim 1,
The step of the processor performing learning,
An image region segmentation method, characterized in that the ground-truth image is also converted into a latent space phenotype, and the converted data is converted into a spike train with respect to time. According to claim 1,
The step of the processor performing learning,
An image region segmentation method, characterized in that the weight using the loss error between the input spike synapse and the output spike synapse is applied and updated. In the computer program stored in a computer-readable recording medium combined with the processor of the computer,
An image region segmentation program, characterized in that the method of any one of claims 1 to 4 is executed. An apparatus for dividing an image region, comprising:
an input unit for receiving image data as an input;
an encoder for converting the image data into a latent space phenotype;
a spike transformation module for converting the data converted into the latent space phenotype or the output data of the learning module into a spike train with respect to time;
an SNN-based learning module for learning the data converted into the spike sequence;
a decoder that converts the data converted into a spike column into a latent space phenotype; and
and an output unit for outputting the data converted into the latent spatial phenotype as region-segmented image data.

Images