KR20200109681A - Data acquisition method for classification of fish species - Google Patents

Abstract

The present invention relates to a data application method for classifying fish species, comprising: a camera provided on the inner water surface; a control unit managing and processing information of the camera; and a terminal capable of outputting information stored in the control unit. The method comprises: a first step of securing fish species and extracting fish images; a second step of generating data by combining the extracted fish image and an underwater environment image with different pixel values; a third step of storing the generated data in the control unit; a fourth step of extracting the fish image on the inner water surface with the camera; a fifth step of generating classification information by classifying the fish species by comparing the fish image extracted by the control unit with the data; a sixth step of storing the classification information in the control unit; and a seventh step of transmitting the classification information to the terminal.

Description

Data acquisition method for classification of fish species}

The present invention relates to a method of applying data for classification of fish species, and a method of collecting data by transforming an image image to improve fish classification performance, and then synthesizing it with an image of an inland water surface using images taken directly for each fish species. It is about.

The Ministry of Environment has been monitoring the exotic species that disturb the ecosystem since 2007, and has been conducting annual surveys on 16 foreign species since 2011. In particular, at the 10th General Assembly of the Parties to the Convention on Biological Diversity in 2010, research on the removal of invasive alien species is being actively conducted both at home and abroad by announcing “Removal of invasive alien species” among the goals for biodiversity conservation. In particular, large mouse bass and bluegill, which are foreign fish species, are known to be the most important factors in reducing the population of native fish in Korea as organisms disturbing the ecosystem in the domestic water surface. Therefore, it is necessary to develop an efficient and reliable system for combating these foreign fish species. Therefore, prior to the development of the entire system for combating foreign fish species, the development of a system for classification of fish species must be preceded.

Since many years ago, interest in machine learning and deep learning has been increasing.In particular, deep learning is a method of machine learning, which is a network model that a computer directly learns from data, classifying images, text, etc. You can learn. This development has developed into a highly practical technology thanks to the use of big data, the development of hardware such as high-performance GPU (Graphic Processing Unit), and the development of recognition algorithms. As various types of media increase, deep learning applications Are being newly created. Among them, CNN (convolutional neural network) developed by Professor LeCun in the late 1990s is widely used by successfully applying it to the image recognition field based on the back propagation learning algorithm by imitating the visual processing process of humans or animals. Yes [Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 1998].

Among these image recognition fields, some cases have been successfully applied to the field of fish species recognition [G. Chen, P. Sun and Y. Shang, 2017; V. A. Sindagi and V. M. Patel, 2018], which classified fish species by fish images taken outside the water or on a boat. In addition, the classification of fish living in the natural environment is difficult to accumulate data on fish of the same species, resulting in many difficulties in learning and testing [M. Sarigul and M. Avci, 2017; H. Qin, X. Li, J. Liang, Y. Peng and C. Zhang, 2016; A. Salman, A. Jalal, F. S., A. Mian, M. Shortis, J. Seager and E. Harvey, 2016]. In addition, general CNNs are showing excellent performance in the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) competition, but the targets to be classified are composed of data with very distinct characteristics such as cars, horses, people, boats, and flowers. However, in the case of fish, all the objects to be classified have a similar streamlined shape, and the environment of the object to be classified shows different characteristics according to the surrounding natural environment in the water.

Korean Patent Registration No. 10-1656635

In order to solve the above-described problem, an object of the present invention is to provide a data application method for classifying fish species with excellent fish species classification performance and a short execution period.

In order to achieve the above object, it is composed of a camera provided on the inland water surface, a control unit for managing and processing information of the camera, and a terminal capable of outputting the information stored in the control unit, and securing fish species to obtain fish images. A first step of extracting the fish image and a second step of synthesizing the extracted fish image and the underwater environment image with different pixel values, and a third step of storing the generated data in a control unit, and the camera A fourth step of extracting a fish image of the inland water surface; a fifth step of generating classification information by classifying a fish species by comparing the fish image extracted by the control unit with the data; and a fifth step of storing the classification information in the control unit It consists of a sixth step and a seventh step of transmitting the classification information to the terminal.

The control unit uses a convolutional neural network, and classifies fish species by comparing the data and fish images.

이며,

는 이미지 합성 비율로

이고,

는 상기 어류 이미지의 픽셀값이며,

는 상기 물속환경 이미지의 픽셀값인 것을 특징으로 한다.In the second step, the pixel value of the data is

Is,

Is the image composition ratio

ego,

Is the pixel value of the fish image,

Is a pixel value of the underwater environment image.

The data application method for classifying fish species according to the present invention has the following effects.

It shows high fish classification performance for fish images similar to the natural environment, and its execution time and learning time are short, making it most suitable for system development to combat foreign fish species.

There is an effect of improving fish classification performance and developing an efficient fish classification system for combating foreign fish species.

Although it is difficult for experts to recognize the exact fish species with the naked eye because of the underwater environment or similar shapes between fish, there is an effect that effective fish classification is possible through the present invention.

1 is a diagram showing the configuration of a data application method for classifying fish species according to the present invention.
2 is a diagram showing a sequence of a data application method for classifying fish species according to the present invention.
3 is a diagram of a hierarchical structure of a control unit in a data application method for classifying fish species according to the present invention.
4 is a diagram showing an example of a method of extracting fish images in the method of applying data for classification of fish species according to the present invention.
5 is a view showing an example of an underwater environment image in the method of applying data for classification of fish species according to the present invention.
6 is a view showing an example of bluegill synthesized data in a data application method for classifying fish species according to the present invention.
7 is a diagram showing a learning process of data set D of data group I in the data application method for classifying fish species according to the present invention.
8 is a view showing the fish species classification performance of data group I in the data application method for classifying fish species according to the present invention.
9 is a view showing an example of accurately classifying fish species among data group I in the data application method for classifying fish species according to the present invention.
10 is a diagram showing an example of incorrectly classifying fish species among data group I in the data application method for classifying fish species according to the present invention.
11 is a view showing a performance evaluation result of data group I in the data application method for classifying fish species according to the present invention.
12 is a view showing the performance evaluation result of data group I in the data application method for classifying fish species according to the present invention.
13 is a diagram showing the number of data in data group II in the data application method for classifying fish species according to the present invention.
14 is a view showing the performance evaluation results of data group II in the data application method for classifying fish species according to the present invention.
15 is a view showing the performance evaluation result of data group II in the data application method for classifying fish species according to the present invention.
16 is a view showing the performance evaluation result of data group III in the data application method for classifying fish species according to the present invention.
17 is a view showing the performance evaluation results of data group III in the data application method for classifying fish species according to the present invention.
18 is a diagram showing the number of training data according to the accuracy of data groups Ⅱ and Ⅲ in the data application method for classifying fish species according to the present invention.

Hereinafter, a preferred embodiment of a data application method for classifying fish species according to the present invention will be described in detail with reference to the accompanying drawings.

The method of applying data for classification of fish species of the present inventor includes a camera 100 provided on the inland water surface, a control unit 200 for managing and calculating information of the camera 100, and outputting information stored in the control unit 200. The first step (S1) of securing fish species and extracting fish images, and the first step (S1) of extracting fish images and the extracted fish images and underwater environment images with different pixel values and synthesized to generate data A second step (S2), a third step (S3) of storing the generated data in the control unit 200, a fourth step (S4) of extracting an image of a fish on the inland water surface with the camera 100, and the control unit A fifth step (S5) of generating classification information by classifying a fish species by comparing the fish image extracted in 200 with the data, and a sixth step (S6) of storing the classification information in the control unit 200; , It consists of a seventh step (S7) of transmitting the classification information to the terminal 300.

First, the camera 100 is provided on the water surface. The camera 100 may be provided on an inland water surface such as a river or a reservoir, and may photograph fish on the inland water surface. In addition, it is preferable that the camera 100 has a waterproof function.

A control unit 200 for managing information of the camera 100 is provided. The controller 200 may transmit and receive various types of information in addition to the information of the camera 100, and may manage and process the information.

Here, the control unit 200 uses a convolutional neural network, also referred to as a CNN (Convolution Neural Network), in the image recognition field that has the advantage of being able to recognize even if the size and position of an object changes by imitating the visual processing process of an animal. It is a deep learning algorithm with the best performance. Therefore, CNN can learn various types of objects and has the advantage of retraining the network by applying the network learned through transfer learning to a new recognition task, enabling high accuracy and fast learning. . As shown in FIG. 3, a general hierarchical structure of a CNN can be confirmed, and can be divided into a feature extraction layer and a classification layer. The feature detection layer uses a convolution filter on the input image to activate the features of the image, eliminates the vanishing gradient of the network, and allows the network to learn quickly. It can be divided into a linear unit layer, a pooling layer that simplifies the output by performing nonlinear downsampling and reduces the number of parameters to be learned by the network. In addition, the classification layer is a node connected after passing through the feature detection layer, and is generally composed of a fully connected layer (FC), and refers to the number of classifications that the network can predict with a K-dimensional vector.

The control unit 200 may use AlexNet, Vgg16, Vgg19, GoodLeNet, etc. having a simple structure and high performance, but is not limited thereto.

A terminal 300 for outputting information stored in the control unit 200 is provided. Here, the terminal 300 may include not only a portable terminal such as a smart phone, a notebook computer, and a personal digital assistant (PDA), but also a terminal capable of wired communication such as a personal computer (PC).

First, in the first step (S1), fish images are extracted by securing fish species.

It is difficult to secure data by using Internet images or moving images for extracting fish images because it takes a lot of time or there is a problem such as copyright. And in the natural environment, fish images appear very different depending on the depth and turbidity in the water, and the field of view is limited depending on the depth and turbidity in the water, so taking a fish image in the natural environment to acquire data also takes a lot of time. And there was a problem that it was very difficult to shoot certain fish species.

Accordingly, as shown in FIG. 4 in detail, a specific fish species was secured in a laboratory environment, and a moving image of the fish was directly captured, and an image of the fish species was extracted based on the captured video.

Then, the second step (S2) generates data. Specifically, the extracted fish image and the underwater environment image are synthesized after having different pixel values to generate the data.

이며,

는 이미지 합성 비율로

이고,

는 상기 어류 이미지의 픽셀값이며,

는 상기 물속환경 이미지의 픽셀값이다. 구체적으로, 픽셀에 덧셈 연산을 하는 경우 이미지의 명암도를 높여서 밝게 만들어주며, 뺄셈 연산을 하는 경우 이미지의 명암도를 낮추어서 어두운 이미지로 만들어준다. 곱셈 연산의 경우 밝은 값을 더욱 밝게 하고 어두운 값은 약간 밝게 만들어 이미지를 보다 선명하게 부각시키며, 나눗셈 연산은 이미지의 선명도를 낮추는 역할을 한다.Here, the pixel value of the data is

Is,

Is the image composition ratio

ego,

Is the pixel value of the fish image,

Is a pixel value of the underwater environment image. Specifically, when an addition operation is performed on a pixel, the brightness of the image is increased to make the image bright, and when the subtraction operation is performed, the image is made dark by lowering the contrast. In the case of the multiplication operation, the bright value is made brighter and the dark value is slightly brighter to highlight the image more clearly. The division operation serves to lower the sharpness of the image.

The fish image and the image of the underwater environment were different from each other to adjust the sharpness, and the data synthesized through this was made very similar to the image taken in the water, which can replace the image taken from the inland water surface. .

In addition, the underwater environment image can be used as an image of all environments occurring in the inland water in order to realize a video image close to the natural environment as much as possible. As shown in FIG. 5, an image of a green algae component, an image of muddy water, and a deep water depth One of the images may be used, but is not limited thereto.

Then, the third step (S3) stores the generated data in the control unit 200. Specifically, the control unit 200 learned the data by using transfer learning.

Then, in the fourth step (S4), an image of a fish on the inner water surface is extracted. Specifically, the camera 100 is used to extract a fish image of an actual inland water environment.

And, in the fifth step (S5), the extracted fish image is compared with the data. Specifically, the fish image extracted by the control unit 200 is compared with the data stored in the control unit 200 to classify a fish species and generate classification information.

The control unit 200 extracts features of data through a local receptive field, share weights, and sub-sampling. The local reception area means that the nodes of the lower layer are not connected to all nodes of the upper layer, but are connected to only some nodes locally, and the weight sharing is a feature point regardless of the position of the pattern by jointly using a convolution filter. Serves to find. In addition, the two-stage extraction serves to shorten the computation time by reducing the number of parameters while saving feature points through a pooling process.

Like other networks, the control unit 200 is largely composed of an input layer, an output layer, and a large number of hidden layers between the two layers, and can be divided into a feature detection layer and a classification layer. The feature detection layer performs convolution, ReLU (Rectified Linear Unit), and pooling processes, and convolution plays a role of activating the features of the image using a convolution filter on the input image. ReLu keeps the image data at positive values so that the network learns quickly so that vanishing gradients do not occur. In addition, pooling simplifies the output by performing nonlinear downsampling and reduces the number of parameters to be learned by the network. The classification layer is a node connected after passing through the feature detection layer, and is generally composed of a fully connected layer (FC), and outputs a K-dimensional vector. K is the number of classes that the network can predict.

Here, the classification information includes classification performance for each fish species, an example of accurately classifying an image of a fish species, an example of incorrectly classifying an image of a fish species, and the like, but is not limited thereto.

Then, the sixth step (S6) stores the classification information in the control unit 200.

Then, the seventh step (S7) transmits the classification information to the terminal 300. The classification result can be checked through the terminal 300.

Hereinafter, the experimental contents of the method of applying data for classification of fish species according to the present invention will be described in detail.

1. Analysis result-1

The experiment was conducted with Data Group I, which consisted of a data set in which fish images captured in a laboratory environment were converted into images similar to the actual inland water surface.

으로 설정하고, 미니배치(Mini Batch)의 크기는 각 네트워크의 성능에 맞게 적절히 조정하였다. 하드웨어 장치로는 Intel i9-7900 3.3GHz CPU와 4개의 NVIDIA GeForce GTX1080Ti를 사용하였다.CNNs for the experiment were evaluated using AlexNet, GoogLeNet, Vgg16, and Vgg19, which have a simple structure and high performance. Each network tried to reduce the learning time by using transfer learning used in Matlab. Adam was used as an optimization method for each network, and the initial learning rate was

And the mini-batch size was properly adjusted to suit the performance of each network. As the hardware device, Intel i9-7900 3.3GHz CPU and 4 NVIDIA GeForce GTX1080Ti were used.

Fish images of Bluegill, Catfish, Common Carp, Crucian Carp, Large Mouth Bass, Mandarin Fish, and Skin Carp for each fish species Was synthesized with the underwater environment image, and in data group Ⅰ, data set A has a value of 0.3, data set B has a value of 0.5, data set C has a value of x 0.7, and data set D has a value of x The value of is 0.8, the value of x is 0.85 in the data set E, the value of x is 0.9 in the data set F, the value of x is 0.925 in the data set G, and the value of x is 0.95 in the data set H And, Figure 6 is an example showing a data set of Bluegill (Bluegill).

For each fish species, 20% of the 5,000 [jang/fish species] generated in the second step were selected as test data, and 56% and 24% were selected as data for learning and validation.

FIG. 7 shows the learning process of AlexNet for Data set D in the data group I. Since the mini-batch size is 1024 and the total number of learning is 450, epochs of 23.56 were performed. After learning, AlexNet's performance evaluation showed a high accuracy of 98.33% for 1000 test data [jang/fish species].

FIG. 8 shows the fish species classification performance of AlexNet, and since the carp is very similar to the crucian carp image, 14 carp images were erroneously recognized as crucian carp images, and 7 crucian carp images were also erroneously recognized as carp images. Therefore, the classification performance of carp was found to be 96.6%, but the accuracy of classification for other fish species was over 97.5%.

9 and 10 show examples of classifying images of crucian carp and carp of AlexNet, and FIG. 9 shows that the image of crucian carp is accurately classified even though it is very similar to the image of carp. 10 is an example in which the image of a carp is incorrectly classified as a large-mouth bass, and the image of a carp that is incorrectly classified is difficult to accurately recognize even fish experts.

FIG. 11 shows mini-batch for 4 types of CCNs, execution time for one image, and classification performance, and FIG. 12 shows a graph of classification performance of CNNs. When a fish classification system is installed in a natural environment, fast execution time is absolutely required, and the execution time for one image is about 4.3 times that of GoogLeNet, about 11.1 times that of Vgg16, and about 11.6 times that of Vgg19. I confirmed this fast.

In the performance evaluation for data group I, CNNs all showed 95% or higher classification performance for Data sets A to F, and Data set G also showed 90% or higher classification performance. In particular, the turbidity of fish images is very high up to data sets D to H, so even people are often mistaken for it. Overall, it was confirmed that the perception of CNN was much higher than that of humans. As a result of the performance evaluation for data group I, it was confirmed that CNNs have excellent performance in accurately learning and classifying even a wide range of fish images that change depending on the light intensity due to sunlight and the depth and turbidity in the natural environment.

2. Analysis Result-2

The experiment was conducted with Data Group II, which consisted of a data set in which fish images photographed in a laboratory environment were converted into images similar to the actual inland water surface, and an appropriate number of data was evaluated.

으로 설정하고, 미니배치(Mini Batch)의 크기는 각 네트워크의 성능에 맞게 적절히 조정하였다. 하드웨어 장치로는 Intel i9-7900 3.3GHz CPU와 4개의 NVIDIA GeForce GTX1080Ti를 사용하였다.CNNs for the experiment were evaluated using AlexNet, GoogLeNet, Vgg16, and Vgg19, which have a simple structure and high performance. Each network tried to reduce the learning time by using transfer learning used in Matlab. Adam was used as an optimization method for each network, and the initial learning rate was

And the mini-batch size was properly adjusted to suit the performance of each network. As the hardware device, Intel i9-7900 3.3GHz CPU and 4 NVIDIA GeForce GTX1080Ti were used.

Fish images of Bluegill, Catfish, Common Carp, Crucian Carp, Large Mouth Bass, Mandarin Fish, and Skin Carp for each fish species Was synthesized with the underwater environment image, and in data group Ⅱ, data having a certain ratio of data sets A to H of data group Ⅰ were composed, and data sets O to S were created by setting the number of data differently as shown in FIG. Data group II was set up.

14 is a diagram showing the performance evaluation result of data group II, and FIG. 15 is a diagram schematically showing this. As shown in FIGS. 14 and 15, it was confirmed that classification performance increased as the number of training data increased. Among CNNs, the performance of Vgg16 and Vgg19 shows better performance than other networks when the number of training data is large, but the performance is rapidly deteriorated as the number of training data decreases. This is because Vgg16 and Vgg19, which have a very large number of parameters compared to AlexNet and GoogLeNet, show difficulty in learning due to insufficient number of training data.

For classification performance of 90% or more by applying CNNs, the number of training data must be set to Data set P (learning: 2,100[length/fish species]) or more. If the classification performance is satisfied above 85%, the training data can be set to about Data set Q (learning: 1,400[jang/fish species]). This does not require tens of thousands of image data because there are few fish species to be classified. If there are many fish species to be classified, the number of fish image data will also increase.

3. Analysis Result-3

The experiment was conducted with Data Group III, which consisted of a data set in which fish images photographed in a laboratory environment were converted into images similar to the actual inland water surface, and an appropriate number of data was evaluated.

으로 설정하고, 미니배치(Mini Batch)의 크기는 각 네트워크의 성능에 맞게 적절히 조정하였다. 하드웨어 장치로는 Intel i9-7900 3.3GHz CPU와 4개의 NVIDIA GeForce GTX1080Ti를 사용하였다.CNNs for the experiment were evaluated using AlexNet, GoogLeNet, Vgg16, and Vgg19, which have a simple structure and high performance. Each network tried to reduce the learning time by using transfer learning used in Matlab. Adam was used as an optimization method for each network, and the initial learning rate was

And the mini-batch size was properly adjusted to suit the performance of each network. As the hardware device, Intel i9-7900 3.3GHz CPU and 4 NVIDIA GeForce GTX1080Ti were used.

Fish images of Bluegill, Catfish, Common Carp, Crucian Carp, Large Mouth Bass, Mandarin Fish, and Skin Carp for each fish species Was synthesized with the underwater environment image, and it is difficult for humans to recognize the images up to the data set D to H of the data group I, and assuming that it is a general natural environment, it is determined that only Data sets A to C will be sufficient. Data sets U to Y were made from fish species images including data up to C randomly, and this was configured equal to the number of data in data group II to set data group III.

As a result of the performance evaluation of data group Ⅱ, it was experimentally confirmed that the higher the number of training data, the higher the classification performance. However, Data Group II contains severe fish images with low data recognition reliability. Therefore, the same experiment as Data Group II was conducted using Data Group III assuming a general natural environment that can be 100% recognized by humans. FIG. 16 is a result of performance evaluation of data group III, and FIG. 17 schematically shows this.

In FIG. 17, the overall data trend is similar to that of FIG. 15, but the overall classification performance is greatly improved. In particular, AlexNet and GoogLeNet performed well over 91.94% for data set X (learning: 700 [field/fish species]). It can be seen that AlexNet, which has a shallow network, is efficient when the number of fish species to be classified is small even with a small number of training data.

When the number of training data is expressed by averaging the classification performance of CNNs for the data sets of FIGS. 14 and 16, the form of an exponential function as shown in FIG. 18 can be obtained. This indicates that the number of training data increases exponentially according to classification performance. Therefore, the number of training data is affected by the number to be classified, accuracy of classification, and reliability of data recognition.

If the data application method for classifying fish species according to the present invention is limited to a classification accuracy of 90% or more, in the case of data group II, it requires 1500 [chapter/fish species] and 700 [chapter/fish species] or more for data group III. Therefore, assuming that the data recognition reliability of Data Group III is 100%, it is appropriate to set it to about 700 [sheets/fish species], which is 100 times the number to be classified, and to set the classification accuracy to 95%, it is approximately 167 times (1170). /Fish species] is required.

As described above, it will be understood that the above-described technical configuration of the present invention can be implemented in other specific forms without changing the technical spirit or essential features of the present invention by those skilled in the art.

Therefore, the embodiments described above are to be understood as illustrative and non-limiting in all respects, and the scope of the present invention is indicated by the claims to be described later rather than the detailed description, and the meaning and scope of the claims and the All changes or modifications derived from the equivalent concept should be interpreted as being included in the scope of the present invention.

100: camera
200: control unit
300: terminal
S1: Fish image extraction
S2: Create data by combining fish images with underwater environment images
S3: Save data
S4: Extract the image of fish in the inland water
S5: Classification of fish species
S6: Save classification information
S7: Deliver classification information to the terminal

Claims (3)

A camera provided on the water surface;
A control unit for managing and calculating information of the camera;
And a terminal capable of outputting information stored in the control unit,
A first step of obtaining fish species and extracting fish images;
A second step of generating data by synthesizing the extracted fish image and the underwater environment image with different pixel values;
A third step of storing the generated data in a control unit;
A fourth step of extracting an image of a fish on the inner water surface with the camera;
A fifth step of generating classification information by classifying a fish species by comparing the fish image extracted by the control unit with the data;
A sixth step of storing the classification information in a control unit;
A data application method for classifying fish species, comprising: a seventh step of transferring the classification information to a terminal. The method of claim 1,
The control unit uses a convolutional neural network, and classifies a fish species by comparing the data with a fish image. The method of claim 1,
In the second step
The pixel value of the data is

Is,

Is the image composition ratio

ego,

Is the pixel value of the fish image,

Is a pixel value of the underwater environment image.

Images