KR102662997B1 - Hyperspectral image classification method and appratus using neural network - Google Patents

Abstract

An image classification device using a neural network is disclosed. An image classification device according to an embodiment includes a receiver that receives an image, extracts spectral features and spatial features by inputting the images into a plurality of sequence blocks, and performs attention on the spectral features and spatial features to generate an enhanced spectrum. It may include a processor that generates an enhanced spectral feature and an enhanced spatial feature, and classifies the image based on the enhanced spectral feature and the enhanced spatial feature.

Description

Hyperspectral image classification method and device using neural network {HYPERSPECTRAL IMAGE CLASSIFICATION METHOD AND APPRATUS USING NEURAL NETWORK}

The examples below relate to a hyperspectral image classification method and device using a neural network.

While RGB has three channels, HSIs (Hyperspectral Images) measure a wide range of wavelengths, over 200 bands. To achieve this, HSI detects complex features that cannot be found in RGB images. RGB images only have spatial features, but HSI has both spectral features and spatial features, showing excellent performance in classification. Therefore, HSI can be classified for similar colors and thus can be applied to a wide range of fields, including remote sensing.

Recently, HSI using machine learning and deep learning has been adopted in areas such as meat quality, land cover mapping, object detection, change detection, and satellite image mapping.

Machine learning involves mathematical models that learn patterns of data classification or linear regression. Deep learning is widely used in various fields such as computer vision or translation. Deep neural networks used in deep learning include multiple layers.

Embodiments may provide image classification technology using a neural network.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An apparatus for classifying images using a neural network according to an embodiment includes a receiver that receives an image; And extracting spectral features and spatial features by inputting the image into a plurality of sequence blocks, and performing attention on the spectral features and spatial features to create enhanced spectral features and enhanced spatial features. ) and may include a processor that classifies the image based on the enhanced spectral features and the enhanced spatial features.

The processor may extract the spectral features by inputting the image into a spectral sequence block among the plurality of sequence blocks, and extract the spatial features by inputting the image into a spatial sequence block among the plurality of sequence blocks.

The spectral sequence block and the spatial sequence block include a memory block including a convolution operation for a hidden state, a convolution operation for an input, and an element-wise sum operation; and a generation block including concatenation and batch normalization operations for the hidden state and the input.

In the image classification device according to an embodiment, the convolution operation for the hidden state and the convolution operation for the input may include a three-dimensional convolution operation.

The processor may generate the enhanced spectral feature by performing a channel global context attention operation on the spectral feature and generate the enhanced spatial feature by performing a position global context attention operation on the spatial feature.

The channel global context attention operation includes a convolution operation, a reshape operation, and a layer normalization operation, and the position global context attention operation includes a convolution operation, a pooling operation, a reshape operation, and a layer normalization operation. May include operations.

The processor may classify the image by generating a one-dimensional vector by connecting the enhanced spectral feature and the enhanced spatial feature, and inputting the one-dimensional vector to a fully connected layer.

An apparatus for classifying images using a neural network according to one embodiment includes a receiver that receives an image; and generating feature information for the image by inputting the image into a plurality of sequence blocks, classifying the image based on the feature information, and the sequence block outputs output based on Equation 1 and Equation 2. It includes a generation block that generates, and Equation 1 above is: And Equation 2 is, , x is the input, h is the hidden state, l is the layer identification element, is batch normalization, ReLU is the activation function, may represent a weight.

An apparatus for classifying images using a neural network according to one embodiment includes a receiver that receives an image; and a memory block that generates feature information about the image by inputting the image into a plurality of sequence blocks, classifies the image based on the feature information, and the sequence block generates an output based on Equation 1. Includes, and Equation 1 is, , x is the input, h is the hidden state, l is the layer identification element, and may represent the weight of the memory block.

A method of classifying an image using a neural network, performed by an image classification device according to an embodiment, includes receiving an image; extracting spectral features and spatial features by inputting the image into a plurality of sequence blocks; generating an enhanced spectral feature and an enhanced spatial feature by performing attention on the spectral feature and the spatial feature; Classifying the image based on the enhanced spectral features and the enhanced spatial features.

A method of classifying an image using a neural network, performed by an image classification device according to an embodiment, includes receiving an image; Generating feature information about the image by inputting the image into a plurality of sequence blocks; and classifying the image based on the feature information, wherein the sequence block includes a memory block or a generation block, and the memory block generates an output based on Equation 1 and Equation 2. It includes at least one of a generation block or a memory block that generates an output based on Equation 3, where Equation 1 is: And Equation 2 is, And Equation 3 above is, , x is the input, h is the hidden state, l is the layer identification element, is batch normalization, ReLU is the activation function, is the weight of the creation block, and may represent the weight of the memory block.

Embodiments may improve classification performance by performing image classification through a neural network with a plurality of branches for extracting spectral features and spatial features.

Embodiments may improve image classification performance by using sequence blocks including memory blocks and generation blocks.

Figure 1 shows a schematic block diagram of an image classification device according to an embodiment.
Figure 2 shows an example of sequential connectivity.
FIG. 3 shows an example of a neural network used by the image classification device shown in FIG. 1.
Figure 4 shows an example of a spectral sequence block of the neural network shown in Figure 3.
Figure 5 shows an example of a spatial sequence block of the neural network shown in Figure 3.
FIG. 6 shows an example of channel global context attention of the neural network shown in FIG. 2.
Figure 7 shows an example of position global context attention of the neural network shown in Figure 2.
FIG. 8 shows a flowchart of the operation of the image classification device shown in FIG. 1.
Figure 9 is a diagram for explaining and comparing image classification performed through an image classification device according to an embodiment with another neural network.
FIGS. 10A to 10C may be diagrams showing the average and variance of classification results for IP, UP, and SV data sets.
FIG. 11 is a diagram illustrating performance according to changes in detailed parameters of an image classification device according to an embodiment.
Figure 12 is a diagram to explain the performance of an image classification device according to changes in conditions.
Figure 13 is a diagram showing the change in model accuracy according to the change in the number of learning data.
Figures 14a and 14b are diagrams showing the number of learnable parameters, learning time, and testing time depending on the type of learning data.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining the embodiments according to the concept of the present invention. They may be implemented in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can make various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, a first component may be named a second component, without departing from the scope of rights according to the concept of the present invention, Similarly, the second component may also be referred to as the first component.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between. Expressions that describe the relationship between components, such as “between”, “immediately between” or “directly adjacent to”, should be interpreted similarly.

The terminology used herein is only used to describe specific embodiments and is not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate the presence of a described feature, number, step, operation, component, part, or combination thereof, and one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

A module in this specification may mean hardware that can perform functions and operations according to each name described in this specification, or it may mean computer program code that can perform specific functions and operations. , or it may mean an electronic recording medium loaded with computer program code that can perform specific functions and operations, for example, a processor or microprocessor.

In other words, a module may mean a functional and/or structural combination of hardware for carrying out the technical idea of the present invention and/or software for driving the hardware.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or limited by these examples. The same reference numerals in each drawing indicate the same members.

Figure 1 shows a schematic block diagram of an image classification device according to an embodiment.

Referring to FIG. 1, the image classification device 10 can perform image classification. The image classification device 10 can classify the image by receiving an image as an input and processing the image using a neural network. An image includes an image of an object created by refraction or reflection of light, and may mean representing the shape of the object using lines or colors. An image may consist of information in a form that a computer can process. For example, the image may include a Hyperspectral Image (HSI).

Neural networks (or artificial neural networks) can include statistical learning algorithms that mimic neurons in biology in machine learning and cognitive science. A neural network can refer to an overall model in which artificial neurons (nodes), which form a network through the combination of synapses, change the strength of the synapse connection through learning and have problem-solving capabilities.

Neurons in a neural network can contain combinations of weights or biases. A neural network may include one or more layers consisting of one or more neurons or nodes. Neural networks can infer the results they want to predict from arbitrary inputs by changing the weights of neurons through learning.

Neural networks may include deep neural networks. Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), and LSTM. (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield) Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network) ), Liquid State Machine (LSM), Extreme Learning Machine (ELM), Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), Those skilled in the art will understand that it may include any neural network, including but not limited to KN (Kohonen Network) and AN (Attention Network).

The image classification device 10 may be implemented with a printed circuit board (PCB) such as a motherboard, an integrated circuit (IC), or a system on chip (SoC). For example, the image classification device 10 may be implemented with an application processor.

Additionally, the image classification device 10 may be implemented within a personal computer (PC), a data server, or a portable device.

Portable devices include laptop computers, mobile phones, smart phones, tablet PCs, mobile internet devices (MIDs), personal digital assistants (PDAs), and enterprise digital assistants (EDAs). , digital still camera, digital video camera, portable multimedia player (PMP), personal navigation device or portable navigation device (PND), handheld game console, e-book ( It can be implemented as an e-book) or a smart device. A smart device may be implemented as a smart watch, smart band, or smart ring.

The image classification device 10 includes a receiver 100 and a processor 200. The image classification device 10 may further include a memory 300.

The receiver 100 can receive an image. Receiver 100 may include a receiving interface. The receiver 100 may output the received image to the processor 200.

The processor 200 may process data stored in the memory 300. The processor 200 may execute computer-readable code (eg, software) stored in the memory 300 and instructions triggered by the processor 200 .

The “processor 200” may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The processor 200 may extract spectral features and spatial features by inputting images into a plurality of sequence blocks. The processor 200 may extract spectral features by inputting an image into a spectral sequence block among a plurality of sequence blocks. The processor 200 may extract spatial features by inputting an image into a spatial sequence block among a plurality of sequence blocks.

The spectral sequence block and the spatial sequence block may include a memory block and a generation block. A memory block can be composed of any time-series network such as RNN, LSTM, or GRU.

The memory block may include a convolution operation on the hidden state, a convolution operation on the input, and an element-wise sum operation. The generation block may include concatenation and batch normalization operations on hidden states and inputs.

The convolution operation on the hidden state and the convolution operation on the input may include a three-dimensional convolution operation.

The processor 200 may generate an enhanced spectral feature and an enhanced spatial feature by performing attention on the spectral feature and spatial feature.

The processor 200 may generate enhanced spectral features by performing a channel global context attention operation on the spectral features, and generate enhanced spatial features by performing a position global context attention operation on the spatial features.

Channel global context attention operations may include convolution operations, reshape operations, and layer normalization operations. The position global context attention operation may include a convolution operation, pooling operation, reshaping operation, and layer normalization operation.

Processor 200 may classify images based on enhanced spectral features and enhanced spatial features. Processor 200 may generate a one-dimensional vector by concatenating the enhanced spectral features and the enhanced spatial features. The processor 200 can classify the image by inputting a one-dimensional vector into a fully connected layer.

The memory 300 may store instructions (or programs) executable by the processor 200. For example, the instructions may include instructions for executing the operation of the processor and/or the operation of each component of the processor.

The memory 300 may be implemented as a volatile memory device or a non-volatile memory device.

Volatile memory devices may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

Non-volatile memory devices include EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM (MRAM), and Conductive Bridging RAM (CBRAM). , FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM (Nanotube RRAM), Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, molecular electronic memory device, or insulation resistance change memory.

Figure 2 shows an example of sequential connectivity.

Referring to FIG. 2, the sequence connection may include a memory block 210 (e.g., a Recurrent Neural Network (RNN) block, an LSTM block, etc.) and a generation block 230. The memory block 210 may include a series of convRNN models with a hidden state h and an input x. The convRNN model uses a convolution operation instead of a fully connected operation. May include sequence-based models (e.g. RNN).

The generation block 230 may include a batch normalization, a Rectified Linear Unit (ReLU) activation function, and a 3-dimensional (3D) convolution layer. The 3D convolution layer can be easily derived by a person skilled in the art through non-patent document 1 (Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift).

Sequence connection may refer to a neural network that improves dense connectivity using layer level information (information about the level (position) of the layer where feature information is generated). The memory block 210 may be composed of a series of convRNN models with a hidden state of h and an input of x.

Dense connectivity is a collective connection that concatenates previous layers, and is a connection structure included in Densenet, while sequence connectivity is a connection structure that includes features of previous layers according to the order of the layers. can be created.

In the example of FIG. 2, x may mean an input and h may mean a hidden state. The output x _l of the generation block 230 can be expressed as Equation 1.

here, can be expressed as Equation 2. is batch normalization ( ), may refer to the configuration of the generation block 230 including ReLU activation and convolution layers.

The output h _l of the memory block 210 can be expressed as Equation 3.

and may mean the weight of convolution layer l of the memory block. h _l can be generated by a convolutional RNN (convolutional RNN)-based model such as RNN, GRU, or LSTM. For example, the processor 200 can generate h _l using Equation 3 through convRNN.

Generally, in the case of RNN, shared weights for blocks can be utilized based on various data lengths and relationships between data. According to one embodiment, the memory block 210 has weights that are not shared (different) for each block. and You can have This is because the length of the entire network is fixed to a specific value (L). non-shared weights and Based on , the memory block 210 can focus more on the order of feature levels.

A network (seqnet) constructed through sequence connections (sequence blocks) according to one embodiment not only reflects feature-level information in skip connections, but also includes the same number of layers. Number of operations compared to Densenet disclosed in non-patent document 2 (Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition) can be saved.

The number of MAC operations (Multiply-accumulate operations) performed for 3D convolution can be calculated based on Equation 4.

is the number of MAC operations, represents the channel, depth, height, and width of the network input feature map, respectively, represents the channel, depth, height, and width of the network output feature map, respectively, represents the depth, height, and width of the kernel, respectively, may represent the depth, height, and width of the stride, respectively.

The number of MAC operations performed in Densenet and the number of MAC operations performed in a network (seqnet) utilizing sequence connections (sequence blocks) according to one embodiment can be expressed as Equation 5.

is the number of MAC operations of a dense block including L layers performed in Densenet, is the number of MAC operations of L sequence blocks performed in a network (seqnet) utilizing sequence connection, is the number of layers, is the initial input channel of Densenet, is the repeated input channel of Densenet, , Is the weight of the memory block 210 ( , ) can be indicated.

When considering 3D convolution computation time regardless of batch normalization and activation function, the kernel size is the same, In this situation, the number of MAC operations performed on L sequence blocks included in Seqnet is measured to be less than the number of MAC operations performed on Dense blocks including L layers included in Densenet ( is 7 or higher). also, In the case of the situation, L sequence blocks (individual blocks included in seqnet) have 10% higher efficiency ( If this is 4), or 25% improved efficiency ( indicates 5). Efficiency can be calculated based on the number of MAC operations performed in Seqnet and Densenet. More specifically, it can be determined that the efficiency of Seqnet has been improved by 10% through the values calculated based on Equation 6.

FIG. 3 shows an example of a neural network used by the image classification device shown in FIG. 1.

FIG. 4 shows an example of a spectral sequence block of the neural network shown in FIG. 3, and FIG. 5 shows an example of a spatial sequence block shown in FIG. 3. Each sequence block shown in FIGS. 3 to 6 can be designed by borrowing the structure of sequence connection described above.

FIG. 6 shows an example of channel global context attention of the neural network shown in FIG. 3, and FIG. 7 shows an example of position global context attention of the neural network shown in FIG. 3.

Referring to FIGS. 3 to 7 , a processor (eg, processor 200 of FIG. 1 ) may perform image classification using a neural network with sequence connections (connection structure included in sequence blocks). The neural network may include two branches for acquiring spatial features and spectral features. Each branch can extract spectral features and spatial features. At the end of the branch, spectral features and spatial features can be merged to create a vector for image (e.g., Hyperspectral Image (HSI)) classification.

The first branch may include a spectral sequence block 310 and the second branch may include a spatial sequence block 330. The spectral sequence block 310 and the spatial sequence block 330 may have the structure of the sequence block described in FIG. 2.

The processor 200 may use sequence connection to consider sequential relationships (layer level information) between layer features in image classification. The output features of the convolution can be considered time t data. The processor 200 may generate gated features including the order of layer information using the hidden state and time t data through a neural network.

The processor 200 can enhance the extracted spectral features and spatial features using the attention block. The attention block may include a channel global context attention block 350 and a position global context attention block 370.

The processor 200 can improve the performance of the neural network by using a global context attention mechanism. The global context attention mechanism may be a mechanism that merges two attention mechanisms (e.g., non-local block and squeeze-excitation block). Simplified non-local blocks capture long-range dependencies, and squeeze excitation blocks can have a light weight computation cost.

3 to 7, if the stride is (1, 1) or (1, 1, 1) and the padding is the same, the stride and padding are omitted in the two-dimensional or three-dimensional convolutional layer. It can be.

A neural network can be composed of three stages. The three stages may include a feature extraction stage, an enhancement stage, and a classification stage. Images input to the neural network may include patch images. For example, a patch image may be part of an HSI with a size of 9×9×1,200. Here, 9 may mean the patch size, and 200 may mean the band size of the input data.

A patch image may be input as two branches, and each branch may correspond to a spectral feature and a spatial feature. The processor 200 may extract spectral features using the spectral sequence block 310 and spatial features using the spatial sequence block 330.

The processor 200 may compress the patch image. For example, the processor 200 converts the patch image into a size of 9×9×97,8 through a valid 1×1×7,8 convolution operation with stride and padding of (1, 1, 2). having It can be compressed. At the same time, the processor 200 fills in zeros. can be created.

refers to the spectral sequence block 310, and Can be used as inputs to the memory block and generation block, respectively. The input of the memory block can be transformed into a size of 9×9×97,8 by two convolutional layers with kernels of size 1×1×1,8 for each input.

The transformed h and x can be input to the element-wise sum and tanh activation functions. The output of the memory block is The shape may have a size of 9×9×97, 8. In the creation block, the input can have a size of 9×9×97, 24 through the concatenate operation. The concatenated inputs can be sequentially fed into operations including batch normalization, ReLU activation function, and 1×1×1, 16 convolution.

The final output of the creation block is silver It may have the same size as .

The spectral sequence block 310 can be repeated L times, and the output after repeating L times is and It can be. At the final stage of the spectrum sequence block, the output can be concatenated to have a size of 9×9×97, 24. The output can be sequentially fed back to batch normalization, ReLU activation function, and 1×1×97, 24 convolutional layers. The spectral feature extraction process and parameters may be as shown in Table 1.

Layer name input size output size kernel size Conv1 (9×9×1,200) (9×9×97,16) (1×1×1, 16) Conv3d -memory block (9×9×97,16) / (9×9×97,8) (9×9×97,8) (1×1×1, 8) ConvRNN -Create blocks (9×9×97,16) / (9×9×97,8)
(9×9×97,24) (9×9×97,24)
(9×9×97,16) -
(1×1×1,16) Concat
BN/ReLU/Conv3d Conv2 (9×9×97, 16)/(9×9×97, 8)
(9×9×97, 24) (9×9×97, 24)
(9×9×1, 24) -
(1×1×97,24) Concat
BN/ReLU/Conv3d

The processor 200 may strengthen spectral characteristics by inputting the output of a 9×9×1, 24-sized spectral sequence block to the channel global context attention block 350. Spectral features can be sharpened through aggregation, transformation and enhancement. The channel global text attention block 350 may be implemented as shown in Table 2.

layer name input size output size kernel size layer operations Feature aggregation ( 9×9×24)( 9×9×1)
(9×9×24)
(1×1×81)/(81×24) (9×9×1)
(1×1×81)
(81×24)
(1×1×24) (1×1, 1)
-
-
- Conv2d
Reshape/Softmax
Reshape
Matrix multiplication Feature conversion ( 1×1×24)( 1×1×24/r)
(1×1×24/r) (1×1×24/r)
(1×1×24/r)
(1×1×24) (1×1, 24/r)
-
(1×1, 24) Conv2d/Dropout
LayerNorm/ReLU
Conv2d Feature Enhancement (9×9×24)/( 1×1×24) (9×9×24) - Element-wise sum

Processor 200 may generate spatial features in a manner similar to spectral features. The patch image has a kernel size of 1Х1Х200, 16, the stride is (1,1,1), and the padding can be input into a valid convolution operation. The processor 200 has an output size of 9Х9Х1, 16 and 9Х9Х1, having a size of 8 can be created simultaneously. The two features can be input into the spatial sequence block 330 similarly to the spectral sequence block. The size of all convolution kernels of the spatial sequence block 330 may be the same as that of the spectral sequence block. and The form can be 9Х9Х1, 16 and 9Х9Х1, 8.

The spatial sequence block 330 may be repeated L times. The spatial sequence block 330 may be implemented as shown in Table 3.

Layer name input size output size kernel size Conv1 (9×9×1,200) (9×9×1,16) (1×1×1, 16) Conv3d -memory block (9×9×1,16) / (9×9×1,8) (9×9×1,8) (1×1×1, 8) ConvRNN -Create blocks (9×9×1,16) / (9×9×1,8)
(9×9×1,24) (9×9×1,24)
(9×9×1,16) -
(1×1×1,16) Concat
BN/ReLU/Conv3d Concat (9×9×1, 16)/(9×9×1, 8) (9×9×1, 24) - Concat

The output may be connected at the end of the spatial sequence block 330 to have a size of 9Х9Х1, 24. The output of size 9Х9Х1, 24 can be input to the position global context attention block 370. The position global context attention block 370 can enhance spatial features and generate an output of the same size as the input. The implementation of the position global context attention block 370 may be as shown in Table 4.

layer name input size output size kernel size layer operations Feature aggregation ( 9×9×24)( 1×1×24)
(9×9×24)
(1×1×24)/(24×81) (1×1×24)
(1×1×24)
(24×81)
(1×1×81) -
-
-
- GlobalAvgPooling
Softmax
Reshape/Transpose
Matrix multiplication Feature conversion ( 1×1×81)( 1×1×81/r)
(1×1×81/r) (1×1×81/r)
(1×1×81/r)
(1×1×81) (1×1, 81/r)
-
(1×1, 81) Conv2d/Dropout
LayerNorm/ReLU
Conv2d Feature Enhancement (9×9×24)/( 9×9×1) (9×9×24) - Element-wise sum

In the feature classification part, the output of the branches can be a vector of size 1×24 by batch normalization, ReLU activation function, and global average pooling operation. The processor 200 merges the outputs to generate a one-dimensional vector with a size of 1 × 48, and passes it through the previous connection layer and soft max function to determine the class of the center pixel in the patch image. can be predicted. The implementation of the classification operation may be as shown in Table 5.

layer name input size output size layer operations Spectral feature conversion (9×9×24) (1×24) BN/ReLU/GlobalAvgPooling Spatial feature transformation (9×9×24) (1×24) BN/ReLU/GlobalAvgPooling connection (1×24)/(1×24) (1×48) Concat classification 1×48 (1×N _Class ) FC/Softmax

Here, N _class may mean the number of predicted classes.

FIG. 8 shows a flowchart of the operation of the image classification device shown in FIG. 1.

Referring to FIG. 8, a receiver (eg, receiver 100 of FIG. 1) may receive an image (810).

The processor 200 may extract spectral features and spatial features by inputting images into a plurality of sequence blocks (830). The processor 200 may extract spectral features by inputting an image into a spectral sequence block among a plurality of sequence blocks. The processor 200 may extract spatial features by inputting an image into a spatial sequence block among a plurality of sequence blocks.

The spectral sequence block and the spatial sequence block may include a memory block and a generation block. The memory block may include an artificial neural network that processes time-series data, such as RNN or LSTM.

The memory block may include a convolution operation on the hidden state, a convolution operation on the input, and an element-wise sum operation. The generation block may include a spectral sequence block and a spatial sequence block may include concatenation and batch normalization operations for hidden states and inputs.

The convolution operation on the hidden state and the convolution operation on the input may include a three-dimensional convolution operation.

The processor 200 may generate an enhanced spectral feature and an enhanced spatial feature by performing attention on the spectral feature and spatial feature (850).

The processor 200 may generate enhanced spectral features by performing a channel global context attention operation on the spectral features, and generate enhanced spatial features by performing a position global context attention operation on the spatial features.

Channel global context attention operations may include convolution operations, reshape operations, and layer normalization operations. The position global context attention operation may include a convolution operation, pooling operation, reshaping operation, and layer normalization operation.

Processor 200 may classify the image based on the enhanced spectral features and enhanced spatial features (870). Processor 200 may generate a one-dimensional vector by concatenating the enhanced spectral features and the enhanced spatial features. The processor 200 can classify the image by inputting a one-dimensional vector into a fully connected layer.

Figure 9 is a diagram for explaining and comparing image classification performed through an image classification device according to an embodiment with another neural network.

Test images used in the experiment may be i) Indian Pines (IP), ii) University of pavia (UP), and iii) Salinas Valley (SV), which are widely used in experiments in related fields. In the experiment, 2% of the IP data set and 0.5% of the UP and SV can be used, and details of individual samples used for training, validation, and testing during the experiment are shown in Figure 9. As shown. More specifically, table 911 may indicate details of an IP data sample, table 912 may indicate details of a UP data sample, and table 913 may indicate details of an SV data sample.

The neural networks with which the operation and performance of the image classification device are compared according to one embodiment are i) support vector machine (SVM) including a radial basis function (RBF) kernel, ii) deep&dense convolutional neural network (DDCNN), and iii) SSRN. (spectral-spatial residual network), iv) SSGCA (spectral and spatial global context attention). During the experiment, patch, batch, and epoch can be set to 9, 64, and 300. The cross entropy function used for learning to evaluate performance is as described in Equation 7.

CE is the cross entropy function, m is the number of ground truth categories, is the class number in the ground truth, can represent the probability of the softmax value from the network. Learning proceeds in the direction of maximizing the probability of the indicator for the correct answer. To prevent local minimum and overfitting, dropout, cosine annealing, and early stopping methods can be applied. During the learning process, randomly selected elements in the network may be set to 0 with probability p (set to 0.5). The cosine annealing method modifies the learning rate based on Equation 8.

is the maximum number of epochs, is the minimum learning rate, may represent the number of epochs performed. By dynamically adjusting the learning rate, the local minimum problem is alleviated and the network accuracy can be improved. Additionally, validation loss is measured at every epoch, and if validation loss does not decrease continuously for 40 epochs, training may be stopped early.

Accuracy, overall accuracy (OA), average accuracy (AA), and kappa coefficient for each class can be used as indicators. OA is the proportion of true positives (TP) in the test sample, AA is the average accuracy per class, and Kappa coefficient can represent the degree of agreement between the ground truth and the predicted class.

FIGS. 10A to 10C may be diagrams showing the average and variance of classification results for IP, UP, and SV data sets.

An image classification device according to an embodiment may utilize RNN and GRU in memory blocks. As can be seen from the experimental results, RNN-Seqnet, whose memory blocks are composed of RNNs, showed the best performance. More specifically, RNN-Seqnet exhibited an OA of 95.03%, showing improved performance of 2.28% over SSGCA, 8.14% over SSRN, 29.98% over SVM, and 43.35% over DDCNN (2% on IP dataset). of training and validation samples).

RNN-Seqnet (memory block composed of RNN) and GRU-Seqnet (memory block composed of GRU) used in image classification devices can reflect feature-level information in the classification process, and SSGCA It can be seen that accuracy is improved compared to . Seqnet shows the highest mean and lowest variance in OA, AA, and Kappa coefficients. Moreover, the gap between the lowest value (78.43%) and the highest value (99.96%) shows that RNN-Seqnet and GRU-Seqnet have the lowest values among the networks. Additionally, it can be confirmed through experiments that Seqnet is the most common and robust network in all data.

As can be seen from the lower part of FIGS. 10A to 10C, it can be confirmed that GRU-Seqnet and RNN-Seqnet used in the image classification device produce classification results that are most similar to the correct answer (high accuracy).

FIG. 11 is a diagram illustrating performance according to changes in detailed parameters of an image classification device according to an embodiment.

Figure 11 (a) shows the OA value according to the change in parameter r. The parameter r may refer to the compression ratio in the global context attention mechanism and may be referred to as the bottleneck ratio. Parameter r has been used previously in FIGS. 6 and 7, and can determine cost efficiency and accuracy. (a) shows the OA value according to the change in parameter r in a situation where the number of sequence blocks L in the network is predetermined to be 4. The image classification device according to one embodiment may determine the parameter r value with the maximum average of OA values in IP, UP, and SV as the optimal parameter r value. Based on the presented experimental results, if the image classification device is composed of RNN-Seqnet, the parameter r value can be determined to be 12, and if the image classification device is composed of GRU-Seqnet, the parameter r value can be determined to be 6.

The image classification device can determine the optimal L value based on the optimal parameter r value determined through (a). (b) shows the OA value calculated according to the change in L value based on the parameter r value determined through (a). The image classification device can determine the L value with the largest average value of OA as the optimal L value. If RNN-Seqnet is used, the image classification device can determine the L value to be 4, and if GRU-Seqnet is used, the L value can be determined to be 4.

Because of the gradient vanishing problem, the performance of GRU-Seqnet can be superior to that of RNN-Seqnet as the layer becomes deeper. The reason why there is no significant difference in FIG. 11 is because the L value is set to 3 to 6, so the layer is not deep enough.

Because RNN-Seqnet extracts more useful features (informative features) than GRU-Seqnet, RNN-Seqnet can show improved performance than GRU-Seqnet in a simple structure.

Figure 12 is a diagram for explaining the performance of an image classification device according to the presence or absence of individual elements.

Figure 12 (a) shows the performance (OA value) of the image classification device depending on the presence or absence of an attention block (channel global context attention block, position global context attention block). Referring to (a), in the case of the SV data set, there is a relatively large difference in the performance of the image classification device depending on the presence or absence of an attention block, but in the case of the IP and UP data sets, the performance difference of the image classifier is relatively small. .

(b) shows the OA value for the case of using a shared weight and the case of using a non-shared weight in the memory block. Generally, in NLP, RNN-based models utilize shared weights. This is because the length of the input data varies greatly. RNNs that utilize shared weights learn relationships between data, so no additional learning is required. This is because the model learns a front-to-back relationship, regardless of data occurring in different locations. This means that non-shared weights can increase the concentration of data location more than shared weights.

Referring to (b), it can be seen that higher performance is achieved when non-shared weights are used for memory blocks.

Figure 13 is a diagram showing the change in model accuracy according to the change in the number of learning data.

When the number of learning data decreases, classification accuracy may decrease, but the degree to which accuracy decreases may vary for each model. Figure 13 (a) shows the accuracy change according to the ratio of training data (ratio of training data to total data) when using the IP data set, and (b) shows the training data ratio when using the UP data set. shows the accuracy change according to , and (c) shows the accuracy change according to the training data rate when using the SV data set. Overall, it can be seen that the accuracy is improving as the ratio of training data among the entire data set increases, but the accuracy decrease rate of GRU-Seqnet and RNN-Seqnet, which are used in image classification devices, is the smallest. Through the above results, it can be seen that GRU-Seqnet and RNN-Seqnet show excellent performance in terms of robustness and generalization compared to other models.

Figures 14a and 14b are diagrams showing the number of learnable parameters, learning time, and testing time according to the type of learning data.

Referring to Figure 14, it can be seen that RNN-Seqnet is the lightest model. Compared to SSGCA, which shows similar OA, RNN-Seqnet contains 18% lower trainable parameters, and 14-33% faster inference is possible for IP and SV data sets.

SSGCA-based DenseAT model designed to compare inference time with Densenet ( and is 16, and the kernel size is 1) has 635,539 weights for the IP data set, which is larger than RNN-Seqnet. The biggest difference between DenseAT and RNN-Seqnet is It's in the 3D convolution behind the block. The channels of DenseAT's input feature map are 80 by stacked layers, and thus 620,880 parameters are required. In contrast, RNN-Seqnet can maintain 24 channels. Because elementwise sum and tanh activation functions spend more time on memory blocks compared to 3D convolution, RNN-Seqnet theoretically exhibits 10% faster inference time, but in actual experiments, it is somewhat slower as shown in the figure. It showed a longer calculation time.

Figure 14b shows the inference time according to the number of layers L of DenseAT and RNN-Seqnet. As the layers get deeper, RNN-Seqnet takes less time compared to DenseAT. When L is 5, RNN-Seqnet takes less computation time than DenseAT for IP and SV data sets. Regarding the UP data set, since the UP data set contains fewer bands, the computation time of element wise sum and tanh activation function dominates compared to the 3D convolution computation time. DenseAT requires L to be greater than 7 to overcome the inference time. When checking the experimental example, it was confirmed that RNN-Seqnet is a relatively light-weight and cost-efficient model compared to SSGCA that utilizes the dense block included in Densenet, and has higher accuracy and faster speed. do.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (10)

In a device for classifying images using a neural network,
A receiver that receives images; and
Extract spectral features and spatial features by inputting the image into a plurality of sequence blocks,
Generating an enhanced spectral feature and an enhanced spatial feature by performing attention on the spectral feature and the spatial feature,
a processor to classify the image based on the enhanced spectral features and the enhanced spatial features;
Including,
Extracting the spectral features by inputting the image into a spectral sequence block among the plurality of sequence blocks,
Extracting the spatial features by inputting the image into a spatial sequence block among the plurality of sequence blocks,
The spectral sequence block and the spatial sequence block are:
A memory block containing a convolution operation on the hidden state, a convolution operation on the input, and an element-wise sum operation; and
A generation block including concatenation and batch normalization operations for the hidden state and the input,
The memory block is,
Generate size-transformed input by two convolutional layers,
By inputting the size-transformed input into the element-wise sum and tanh activation functions, an output is generated,
The creation block is,
Apply the concatenate operation to the input to create a concatenated input,
An image classification device that generates output by sequentially feeding concatenated inputs into batch normalization, ReLU activation functions, and convolution operations.
delete delete According to paragraph 1,
The processor,
Generating the enhanced spectral features by performing a channel global context attention operation on the spectral features,
Generating the enhanced spatial features by performing a position global context attention operation on the spatial features,
Image classification device.
According to paragraph 4,
The channel global context attention operation is,
Includes convolution operation, reshape operation, and layer normalization operation,
The position global context attention operation is,
Including convolution operation, pooling operation, reshaping operation and layer normalization operation,
Image classification device.
According to paragraph 1,
The processor,
generating a one-dimensional vector by connecting the enhanced spectral features and the enhanced spatial features,
Classifying the image by inputting the one-dimensional vector into a fully connected layer
Image classification device.
In a device for classifying images using a neural network,
A receiver that receives images; and
Generating feature information about the image by inputting the image into a plurality of sequence blocks,
Classify the image based on the feature information,
The sequence block is,
Apply the concatenate operation to the input to create a concatenated input,
a generation block that sequentially inputs the concatenated inputs into batch normalization, ReLU activation functions, and convolution operations to generate output;
The creation block is
Generate output based on Equation 1 and Equation 2,
Equation 1 above is:

ego,
Equation 2 above is:

, x is the input, h is the hidden state, l is the layer identification element, is batch normalization, ReLU is the activation function, represents the weight, image classification device.
In a device for classifying images using a neural network,
A receiver that receives images; and
Generating feature information about the image by inputting the image into a plurality of sequence blocks,
Classify the image based on the feature information,
The sequence block is,
Generate size-transformed input by two convolutional layers,
A memory block that generates an output by inputting the size-transformed input to an element-wise sum and tanh activation function,
The memory block is,
Generate output based on Equation 1,
Equation 1 above is:

ego,
x is the input, h is the hidden state, l is the layer identification element, and represents the weight of the memory block, an image classification device.
In a method of classifying images using a neural network, performed by an image classification device,
receiving an image;
extracting spectral features and spatial features by inputting the image into a plurality of sequence blocks;
generating an enhanced spectral feature and an enhanced spatial feature by performing attention on the spectral feature and the spatial feature;
Classifying the image based on the enhanced spectral features and the enhanced spatial features.
Including,
The step of extracting the features is,
Extracting the spectral features by inputting the image into a spectral sequence block among the plurality of sequence blocks,
Extracting the spatial features by inputting the image into a spatial sequence block among the plurality of sequence blocks,
The spectral sequence block and the spatial sequence block are:
A memory block containing a convolution operation on the hidden state, a convolution operation on the input, and an element-wise sum operation; and
A generation block including concatenation and batch normalization operations for the hidden state and the input,
The memory block is,
Generate size-transformed input by two convolutional layers,
By inputting the size-transformed input into the element-wise sum and tanh activation functions, an output is generated,
The creation block is,
Apply the concatenate operation to the input to create a concatenated input,
An image classification method that produces output by sequentially inputting concatenated inputs into batch normalization, ReLU activation functions, and convolution operations.
In a method of classifying images using a neural network, performed by an image classification device,
receiving an image;
Generating feature information about the image by inputting the image into a plurality of sequence blocks; and
Classifying the image based on the feature information
Including,
The sequence block is,
Contains a memory block or generation block,
The step of extracting the feature information is,
Extract spectral features by inputting the image into a spectral sequence block among the plurality of sequence blocks,
Extracting spatial features by inputting the image into a spatial sequence block among the plurality of sequence blocks,
The spectral sequence block and the spatial sequence block are:
A memory block containing a convolution operation on the hidden state, a convolution operation on the input, and an element-wise sum operation; and
A generation block including concatenation and batch normalization operations for the hidden state and the input,
The memory block is,
Generate size-transformed input by two convolutional layers,
By inputting the size-transformed input into the element-wise sum and tanh activation functions, an output is generated,
The creation block is,
Apply the concatenate operation to the input to create a concatenated input,
The concatenated inputs are sequentially fed into batch normalization, ReLU activation function, and convolution operations to generate output;
The memory block is,
It includes at least one of a generation block that generates an output based on Equation 1 and Equation 2, or a memory block that generates an output based on Equation 3,
Equation 1 above is:

ego,
Equation 2 above is:

ego,
Equation 3 above is:

ego,
x is the input, h is the hidden state, l is the layer identification element, is batch normalization, ReLU is the activation function, is the weight of the creation block, and represents the weight of the memory block, image classification method.