KR101743270B1 - The method for separating and recognizing individual plankton using deep learning in a clumped or dispersed microscopic plankton image - Google Patents

Abstract

A method for separating and recognizing individual plankton using deep running in a microscopic image in which a plurality of plankton are coagulated or scattered according to the present invention is a method for detecting and detecting individual plankton by detecting the position of individual plankton from a coherent or scattered microscopic image of a plurality of plankton Separating plankton from the plankton and constructing a CNN (Convolutional Neural Network) to recognize the type of plankton separated; After inputting the input image of plankton at the input of the CNN, the type of plankton output from the output terminal of the CNN, the positional information of the plankton in the input image, and the actual type of plankton in the input image desired to be output from the output terminal of CNN In order to minimize the difference between the actual expected position of the plankton in the input image and the output expected value of the planet, the type of plankton output from the output of CNN, and CNN Learning the CNN by adjusting a weight, which is a variable value; Inputting a photographed image in which a plurality of plankton aggregates or scattered in the CNN; And recognizing the position of the plankton and the type of plankton in the input image by using the CNN, and then displaying the position of the plankton in the input image in the form of a square box, describing the types of plankton recognized on the square box, .

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for separating and recognizing individual plankton using deep running in a microscopic image in which a large number of plankton are coagulated or scattered,

The present invention relates to a method for separating and recognizing individual plankton using deep running in a microscopic image in which a large number of plankton are aggregated or scattered.

In general, plankton is at the bottom of the natural food chain and plays a very important role in marine ecosystems, and the plankton has harmful species and species.

In order to keep the marine ecosystem healthy, it is important to monitor the distribution of species to plankton and issue alarms when necessary.

Monitoring of the plankton can be performed manually or automatically.

Typically, manual approaches that rely on human expertise are tedious, time consuming, and error prone.

Thus, a number of automatic approaches with recognition capabilities for plankton have been proposed.

Until recently, the automatic recognition method of plankton is mainly based on pattern recognition or feature extraction based on classical machine learning method.

At this time, features used are plankton shape, texture, invariant moment, etc. SVM (Support Vector Machine), ensemble classifier, and multi-layer neural network (NN) are used for the recognition of plankton.

However, only a few methods employ in-depth learning methods that outperform plankton-aware performance of existing approaches.

However, almost all in-depth learning methods assume that there is only one plankton in a single image, so it can not be recognized if there are multiple cohorts of the same species plankton or several scattered plankton species, There was a problem that the position of the plankton could not be found.

As a research related to the present invention, Luo et al. Proposed an active learning approach using a Shadow Image Practice Profiling Evaluation Recorder (SIPPER II) image.

They extracted 49 features first, including normal / weighted moments, Fourier descriptors, and texture features, and applied a multi-class SVM for recognition.

The number of samples tested is only 8,440 in 5 species, and the recognition accuracy is 61%.

Hu et al. Used co-occurrence matrices as features from images captured by the Video Plankton Recorder (VPR).

SVM was used as a classifier to recognize 20,000 examples of 7 plankton, and their accuracy was reported to be about 72%.

In addition, Blaschko et al. Used a single or ensemble classifier to extract various features such as moments, textures, contours, and differential features in plankton images captured with a low resolution FlowCam image sensor and for plankton recognition.

Blaschko et al. Tested 982 cases of 13 plankton according to the ensemble method. The test results showed accuracy of 53% to 73%.

Cowen et al. Used a high-resolution line scan camera system called ISIIS to obtain images.

It was assumed that multiple organisms could exist in a single image, and objects were detected by mixing Gaussian background models.

Next, a shape histogram, object rigidity, Hu moment up to third order, Fourier descriptor, and round projection descriptor were extracted.

Finally, a multi-class SVM was applied to recognition.

Cowen et al. Tested 1,110 5 plankton species and showed 61% to 97% accuracy depending on the phytoplankton taxa.

Schulze et al. Used the Sobel operator and histogram normalization in preprocessing and Region growing method to partition the plankton from the plankton images captured by the FlowCam image sensor.

Next, Schulze et al. Extracted elliptic Fourier descriptors and co-occurrence matrices, directional histograms, moments, and rotation invariant local binary patterns.

Finally, we used a two - level multilayer neural network. In the first stage, the non plankton and plankton were separated and in the second stage recognition was performed.

The Schulze et al. Used 1,418 plankton images of 10 plankton images and recorded an average of 94.7% accuracy.

This system was developed as an integrated system known as Plankton Vision.

More than 1,000 teams participated in the contest for the plankton classification held at the National Data Science Bowl (NDSB) from December 2014 to March 2015, and finally the "Deep Sea" team won.

There were 30,000 images of 121 kinds of plankton, and it was a task to sort them accurately.

Most top ranking teams adopted an in-depth learning structure, and Deep Sea reported a log loss of 0.595671.

However, almost all of the previous studies have had the problem that many plankton, which may occur frequently in a real environment, are not allowed to flocculate or scatter.

On the other hand, as a prior art of the present invention, "Method for measuring phytoplankton population using digital image processing technique" of the patent registration number "10-0558613" was filed and registered. The method for measuring phytoplankton population using the digital image processing technique A method for measuring the number of phytoplankton contained in seawater, comprising: preparing a measurement composition by filling seawater containing phytoplankton into a tank having a predetermined size; Irradiating planar light into the measurement composition using a laser and a cylindrical lens; Capturing an area of a predetermined size formed in the planar light with a CMOS camera to acquire a digital image, and transmitting the acquired digital image to a computer system via a main board; Correcting distortion of the digital image obtained using the computer system, removing the image noise through filtering, and then converting the digital image into a measurable image through a preprocessing process for detecting an outline; And extracting a volume of a predetermined size from the converted image to measure the number of phytoplankton.

Korea Patent Registration No. 10-0558613 (Mar. 13, 2006) Korean Patent Publication No. 10-2015-0137047 (Aug. 2015)

Accordingly, it is an object of the present invention to provide a method for separating and recognizing individual plankton by using deep running in a microscopic image in which a large number of plankton are scattered or scattered.

In order to accomplish the above object, the present invention provides a method for separating and recognizing individual plankton by using deep running in a microphotograph image in which a plurality of plankton are flocculated or scattered, comprising the steps of separating and recognizing individual plankton from a microphotograph of aggregated or scattered plankton, Separating the individual plankton by detecting the position of the plankton and constructing a CNN (Convolutional Neural Network) for recognizing the kind of the separated plankton; After inputting the input image of plankton at the input of the CNN, the type of plankton output from the output terminal of the CNN, the positional information of the plankton in the input image, and the actual type of plankton in the input image desired to be output from the output terminal of CNN In order to minimize the difference between the actual expected position of the plankton in the input image and the output expected value of the planet, the type of plankton output from the output of CNN, and CNN Learning the CNN by adjusting a weight, which is a variable value; Inputting a photographed image in which a plurality of plankton aggregates or scattered in CNN; And recognizing the position of the plankton and the type of plankton in the input image by using the CNN, and then displaying the position of the plankton in the input image in the form of a square box, describing the types of plankton recognized on the square box, .

The method of separating and recognizing individual plankton using the deep run in a microscopic image of a plurality of plankton coagulated or scattered according to the present invention comprising the steps of the present invention is a method of separating and recognizing individual plankton from a coherent or scattered input image, It is possible to accurately determine the health state of the ocean or fresh water from which the input image is captured.

1 is a view for explaining CNN (Convolutional Neural Network) applied to the present invention,
2 is a flow chart of the present invention,
3A to 3C are diagrams for explaining a convolution layer,
4 is a view for explaining a Max pooling layer,
5 is a table listing 949 samples of 18 species of noxious plankton species, 15 species of noxious plankton species used in the experiment of the present invention,
6a and 6g show the positions of plankton included in the input image when the coherent or scattered images of a plurality of plankton are input to the present invention, .

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

A method of separating and recognizing individual plankton using a deep run in a microscopic image in which a plurality of plankton are dispersed or scattered according to the present invention is a method of separating and recognizing individual plankton as shown in FIGS. 1 to 2, (S1) of constructing a CNN (Convolutional Neural Network) for separating individual plankton by detecting the position of individual plankton from the image and recognizing the kind of the separated plankton; After inputting the input image of plankton at the input of the CNN, the type of plankton output from the output terminal of the CNN, the positional information of the plankton in the input image, and the actual type of plankton in the input image desired to be output from the output terminal of CNN In order to minimize the difference between the actual expected position of the plankton in the input image and the output expected value of the planet, the type of plankton output from the output of CNN, and CNN A step (S2) of learning CNN by adjusting a weight (Weight) which is a variable value; (S3) of inputting a photographed image in which a plurality of plankton aggregates or scattered in CNN; And recognizing the position of the intracellular plankton and the type of plankton by the CNN, then displaying the position of the plankton in the input image in a rectangular box form, writing the recognized types of plankton on the rectangular box, ).

In step (S1) of constructing a CNN (Convolutional Neural Network) for separating individual plankton and recognizing the kinds of plankton separated by detecting the positions of individual plankton from the micrograph images in which the plurality of plankton coagulated or scattered, The CNN includes a convolution layer for applying a convolution filter mask to an input plankton image and converting the input plankton image into a feature map that can be processed by a computer, A pooling layer for reducing a data size of a feature map outputted through a convolution layer and a convolution layer and a pooling layer, In addition to recognizing plankton from the summarized information, a pulley connected to detect the position of plankton in the input image It includes language (Fully connected layer).

The convolution filter mask is characterized in that it can grasp various types of plankton species in order to grasp the type of plankton from the plankton image and the position of the plankton in the input image.

내지

크기의 컨볼루션 필터 마스크(Convolution filter mask)를 이용하여 CNN으로 입력된 플랑크톤 영상에서 특징을 추출하고, 추출된 특징을 피처 맵(Feature map)으로 출력한다.As shown in FIGS. 3A to 3C, the convolution layer includes a plurality of sub-

To

Size convolution filter mask to extract features from the plankton image input to the CNN and output the extracted features to a feature map.

The number of the convolution layer is 13 or more, and the number of the feature maps is 64 to 1024.

The convolution filter mask extracts the features of the plankton from the image of the input plankton.

If the convolution filter mask has a characteristic that the convolution filter mask has a strong characteristic, the convolution filter mask has a large value, whereas if the convolution filter mask has no characteristic among the input images including plankton, '.

On the other hand, when the input image captured by the plankton is extracted through the convolution layer to some extent, CNN that grasps the type and location of the plankton in the input image need not judge with all the extracted data.

For example, we can identify objects by looking at high-resolution pictures, but even if we have small pictures, we can judge what the contents of the pictures are.

Therefore, artificially reducing the size of the extracted feature map is called sub-sampling or pooling.

The pooling includes Max pooling, Average pooling, and L2-norm pooling. In the pooling layer of the present invention, a Max pooling layer and an Averager pooling layer Average pooling layer) can be used.

피처맵(Feature map)을 스트라이드(Stride)가 2인

풀링 필터(Pooling filter)를 사용할 경우, 총

개의 출력 값이 생기는데, 여기서, 상기 맥스 풀링 레이어(Max pooling layer)는

풀링 필터 안의 4개 파라미터(Parameter) 값들 중 최대(Max) 값 1개를 선택하고 나머지 3개 파라미터 값은 버림으로써 피처맵(Feature map)의 크기를 줄여준다.The pooling layer reduces the size of the feature map of the convolution layer. For example,

The feature map is set to a value of 2 for the stride

When using a pooling filter,

The output of the Max pooling layer is a value of < RTI ID = 0.0 >

The size of the feature map is reduced by selecting one of the four parameter values in the pooling filter and discarding the remaining three parameter values.

풀링 필터 안의 4개 파라미터(Parameter) 값들을 평균 낸 평균(Average) 값 1개를 선택함으로써 피처맵(Feature map)의 크기를 줄여준다.Also, the Average pooling layer

The size of the feature map is reduced by selecting one average value obtained by averaging the four parameter values in the pooling filter.

The Max pooling will be described in more detail with reference to FIG.

의 크기로 잘라낸 다음, 잘라낸

크기의 이미지 영역에서 가장 큰 값을 뽑아내는 방법이다.The Max pooling includes a feature map

, And then cut

This is the method of extracting the largest value in the size image area.

피처맵(Feature map)에서

맥스 풀링 필터(Max pooling filter)를 사용하고, 스트라이드(Stride)는 2로 설정하여 맥스 풀링 필터가 피처맵(Feature map) 상에서 우측으로 2칸 하부로 2칸씩 이동하면서 맥스 풀링(Max pooling)을 처리하는 예인데, 좌측 상단에서는 '6'이 가장 큰 값이기 때문에 '6'을 뽑아내고, 우측 상단에는 2,4,7,8 중 '8'이 가장 크기 때문에 '8'을 뽑아낸다.4,

In the feature map

The Max pooling filter is used and the Stride is set to 2 so that the Max Pooling filter moves Max 2 places to the right by 2 spaces on the Feature map to process Max pooling 6 'is extracted because' 6 'is the largest value in the upper left corner, and' 8 'is extracted in the upper right corner.

The Max pooling represents other feature values with the largest feature value.

The Max pooling is advantageous in that the size of the entire data is reduced, so that computing resources to be computed are reduced.

In the present invention, the number of the pooling layers is four or more. In the pooling layer, a pooling filter is a stride which is a pixel interval in which a pooling filter is moved to a right pixel or a lower pixel on a feature map. (Stride) is set to 2, and when the pulling filter finishes filtering in one section of the feature map, it is shifted by 2 pixels to the right or lower side of the feature map.

The Fully connected layer may be composed of two or more.

The Fully connected layer is a layer of the same type as a conventional neural network in which all the input nodes are connected to all of the output nodes and the computer is actually connected to the pulley connected layer layer) to learn feature data.

The Fully connected layer recognizes the type of plankton by putting the feature values of the plankton extracted from the convolution layer and the pooling layer into the existing neural network, Locate the plankton in the image.

In addition, a Softmax function may be used at the output end of the Fully connected layer to display a probability value according to the type and position of each plankton with respect to the plankton included in the input image.

In addition, the present invention applies a leaky ReLU (Rectified Linear Unit) activation function to a feature map extracted by applying a convolution filter mask.

The reason for using the leaky ReLU activation function is that the back-propagation algorithm method is used because it is difficult to learn as the neural network becomes deeper in CNN. When the Sigmoid function is used as the activation function, The back-propagation algorithm may not work properly (Gradient Vanishing: use the leaky ReLU activation function to remove the forward value when forwarding the result backward from CNN).

After inputting the input image of plankton at the input of the CNN, the type of plankton output from the output terminal of the CNN, the positional information of the plankton in the input image, and the actual type of plankton in the input image desired to be output from the output terminal of CNN In order to minimize the difference between the actual expected position of the plankton in the input image and the output expected value of the planet, the type of plankton output from the output of CNN, and CNN In step S2, CNN is learned by adjusting a weight, which is a variable value, by using CNN learning using supervised learning and learning data composed of an input value and an output expected value. Proceed with learning.

After inputting the input image of plankton at the input of the CNN, the type of plankton output from the output terminal of the CNN, the positional information of the plankton in the input image, and the actual type of plankton in the input image desired to be output from the output terminal of CNN In order to minimize the difference between the actual expected position of the plankton in the input image and the output expected value of the planet, the type of plankton output from the output of CNN, and CNN The step S2 of learning the CNN by adjusting the weight, which is a variable value, is performed by applying an input image photographed with plankton to the CNN input terminal, determining the type of plankton output from the output terminal of the CNN, And the expected output value to be output from the CNN output terminal The difference between the plankton output from the CNN output terminal and the output of the plankton in the input image and the output expected value is referred to, Changing a parameter (weight, Weight) value corresponding to a net of CNN; And another plankton are repeatedly applied to the CNN so that the kind of plankton output from the output end of the CNN and the result of locating the plankton in the input image converge to the expected output value and the actual output value and the output expected value When the difference is minimized, it includes the step of finishing CNN learning.

The output expected value is the exact type of the actual plankton included in the captured input image of the plankton inputted to the input terminal of the CNN and the exact position of the plankton in the input image.

As a result, as shown in FIG. 5, 949 samples of 18 species of three kinds of noxious plankton species and 15 species of noxious plankton species were applied to the present invention, and as a result, 949 all images 100% recognition rate and almost 100% position detection rate.

In addition, applying the present invention to a flow cytometer and a microscope (FlowCam) image, almost perfect results were obtained.

Here, the color box shown in Figs. 6a to 6g shows the position of the detected individual plankton, and the name on the box is the species name for the recognized plankton.

The method of separating and recognizing individual plankton using the deep run in a microscopic image of a plurality of plankton coagulated or scattered according to the present invention comprising the steps of the present invention is a method of separating and recognizing individual plankton from a coherent or scattered input image, It is possible to accurately determine the health state of the marine or fresh water in which the input image is captured because the individual plankton types can be recognized along with the location.

Claims (4)

(S1) of constructing a CNN (Convolutional Neural Network) for separating individual plankton and recognizing the kinds of plankton separated by detecting the positions of individual plankton from a plurality of plankton coagulated or scattered microscopic images;
After inputting the input image of plankton at the input of the CNN, the type of plankton output from the output terminal of the CNN, the positional information of the plankton in the input image, and the actual type of plankton in the input image desired to be output from the output terminal of CNN In order to minimize the difference between the actual expected position of the plankton in the input image and the output expected value of the planet, the type of plankton output from the output of CNN, and CNN A step (S2) of learning CNN by adjusting a weight (Weight) which is a variable value;
(S3) of inputting a photographed image in which a plurality of plankton aggregates or scattered in CNN;
And recognizing the position of the intracellular plankton and the type of plankton by the CNN, then displaying the position of the plankton in the input image in a rectangular box form, writing the recognized types of plankton on the rectangular box, ) In which a plurality of plankton are coagulated or scattered. The method of separating and recognizing individual plankton using deep running.
The method according to claim 1,
In step (S1) of constructing a CNN (Convolutional Neural Network) for separating individual plankton and recognizing the kinds of plankton separated by detecting the positions of individual plankton from the micrograph images in which the plurality of plankton coagulated or scattered,
The CNN includes a convolution layer for applying a convolution filter mask to the input plankton image and converting the input plankton image into a feature map that can be processed by a computer,
A pooling layer for reducing a data size of a feature map outputted through the convolution layer,
And a pulley connected layer for recognizing the plankton from the information abstracted and summarized by the convolution layer and the pooling layer and for recognizing the type of plankton and detecting the position of the plankton in the input image wherein the plankton is separated and recognized using deep running in a microphotograph image in which a plurality of plankton are aggregated or interspersed.
3. The method of claim 2,
The convolution layer

To

The features of the plankton image input to CNN are extracted by using a convolution filter mask of size,
The extracted feature is output as a feature map,
The convolution layer includes 13 or more convolution layers,
Wherein the number of feature maps is from 64 to 1024. 14. A method for separating and recognizing individual plankton using deep running in a coherent or scattered microscope image.
3. The method of claim 2,
The number of the pooling layers is four or more,
In the pooling layer, a pooling filter sets a stride, which is a pixel interval, to be shifted to a right pixel or a lower pixel on the feature map to 2, so that the pooling filter sets one of the feature maps (2) pixels are shifted to the right or the lower side of the feature map when the filtering is finished in the segment. The method of separating and recognizing the individual plankton by using deep running in a micrograph image in which a plurality of plankton are aggregated or scattered .

Images