KR102211078B1 - method and device for classification of parrot - Google Patents

Abstract

Disclosed is a method and apparatus for determining parrot species. Way
Parrots collect a lot of image data for each type of parrot, transform the image data for each type to form type transformation data, prepare training data by collecting image data and transformation data, and train the training data with a convolutional neural network technique. Generate a model file trained for each type, convert the model file into a final model file for a portable terminal device, mount it on the portable terminal device, and obtain comparative image data from the parrot to be identified using a photographing device provided in the portable terminal device. Then, a fully connected layer is formed through a convolutional neural network technique using the trained weight on the comparison image data, and an output layer containing probability information of the parrot is formed using a multi-classification activation function for the fully connected layer.

Description

Method and device for classification of parrot based on image processing

The present disclosure relates to a method and apparatus for separating species of parrots, and more particularly, to a method and apparatus for classifying parrot species by image processing.

Pets that contain a variety of species include parrots. Parrots belong to the order of parrots, and this order includes the family of about 372 species, the Crown parrot family, which includes the cockatiel, and the New Zealand parrot family, which includes Keaamphu kapapo. These various species of parrots are very difficult to classify, and classification itself is sometimes difficult without expert help.

The present disclosure provides a method and apparatus for easily determining the species of a parrot using a portable terminal device.

The present disclosure discloses a method and apparatus for quickly determining the species of a parrot in real time using a convolutional neural network technique.

Method for determining the species of parrot according to the present disclosure:

Collecting a plurality of image data for each type of parrot;

Transforming the type image data to form type modification data;

Preparing training data by collecting the image data and transformation data;

Generating a model file including weights trained for each parrot species by performing training on the training data using a convolutional neural network technique;

Converting the model file into a final model file for a portable terminal device and mounting it on the portable terminal device;

Obtaining comparison image data from a parrot to be determined using a photographing device provided in the portable terminal device;

Forming a fully connected layer through a convolutional neural network technique using the trained weight on the comparison image data;

And forming an output layer containing information of the parrot using a multi-classification activation function for the fully connected layer.

According to one or more embodiments, the convolutional neural network technique may include at least one of a pooling process and a rectification process.

According to one or more embodiments, after convolution is performed on the training data, a rectification process may be performed.

According to one or more embodiments, a softmax algorithm may be applied as a multiple classification activation function.

According to one or more embodiments, the step of forming type deformation data from the type image data comprises changing the width or height of the type image, expanding or reducing it, and forming the deformation data by flipping in a horizontal or vertical direction. can do.

Parrot species determination device according to the present disclosure:

A memory for storing the final model file obtained by the method;

A camera for obtaining comparison image data from the parrot to be determined;

An output layer containing information of the parrot using a convolutional neural network technique using the trained weight on the comparison image data acquired by the imaging device to form a fully connected layer, and using a multi-classification activation function for the fully connected layer A processor forming a;

It may include a; display for outputting the parrot information by the processor.

In the parrot species determination apparatus according to one or more embodiments, the processor may apply a softmax algorithm as a multi-classification activation function.

According to one or more embodiments, in forming the type deformation data from the type image data, the processor converts the deformation data through a change in the width or height of the type image, enlargement or reduction, or flip in a horizontal or vertical direction. Can be formed.

1 is a schematic flowchart of a method for determining a parrot species according to one or more embodiments.
2 is a flowchart illustrating a process of obtaining data and preprocessing in a method for determining a parrot species according to one or more embodiments.
3 is a flowchart of model generation and training in a method for determining a parrot species according to one or more embodiments.
4 illustrates image data obtained by direct photographing of a parrot in a method for determining a parrot species according to one or more embodiments.
5 illustrates image data secured by web crawling in a method for determining parrot species according to one or more embodiments.
FIG. 6 is a view comparing circular image data and transformed image data obtained by transformation in a method for determining parrot species according to one or more embodiments.
7 to 9 illustrate a result of probabilistic determination of a parrot by an apparatus for determining a parrot species according to one or more embodiments.
10 illustrates a schematic configuration of an apparatus for determining a parrot species according to one or more embodiments.

Hereinafter, a method and apparatus for determining a parrot species according to one or more embodiments will be described with reference to the accompanying drawings.

The method for determining parrot species according to the present invention includes the steps as shown in FIG. 1.

The first step is a data collection and pre-processing step for training, and the second step is a model generation and training using data, a third step; Converting and loading the trained model for a portable terminal device, driving a parrot species determination unit (app) for determining the parrot species, and applying image data obtained by photographing a parrot to be determined in real time to the model It includes the step of determining the species of the parrot.

Specifically, the above steps may be subdivided as follows.

<Step 1: Data collection and pre-processing>

1. Video information collection

Image files of various types of parrots can be collected through internet search as video information for deep learning. After searching for images through the Internet, they can be obtained in bulk through crawling or scraping. In addition, an image file (image information) may be obtained by actually photographing one species or a plurality of species of parrots in various directions and postures. Meanwhile, in acquiring an image file from an actual parrot using a camera, a moving parrot may be photographed in various postures for a certain period of time to obtain a video, and a plurality of image files may be acquired from this video in units of frames. However, in this case, a large number of image files are obtained for one species, and if only such image files are used for training, there is a possibility of overfitting. Therefore, in obtaining a video, it is necessary to secure a video selected for various species.

2. Video information expansion

The image information acquired by collection and photographing in the above process may be insufficient as image information for training sufficient in its amount. Therefore, it is necessary to secure sufficient image information for more effective deep learning (model training). To this end, in the present embodiment, a raw image file (information) obtained in the above process is transformed to generate modified image information, and this is used as training image information.

Transformation of an image file includes resizing or rescale, e.g. vertical or horizontal width reduction or expansion, image shift in vertical or horizontal direction, image zoom-in (zoon-n) or zoom-out (image). This may include vertical or horizontal flip, intentional twist or distortion of the image, and the like.

3. Video information preprocessing

The images secured in the above process can have various sizes different from each other, and the size is adjusted to a certain size. For example, all image files are resized to a size of 224*224 vertical*horizontal pixels. The specific size of these images can vary depending on the training module used.

<Step 2: Create and train a training model>

The model as an object for training has a multidimensional structure of CNN (Convolution Neural Network). CNN has multiple layers (layers). Model formation is obtained by declaring a model (object) by programming techniques and by accumulating layers for the model.

This CNN includes the process of feature extraction and classification, and includes known components as follows.

1) Input Layer

The input layer is a multidimensional tensor that stores the information of the raw image. This tensor includes the image number and the pixel value of each of the B/W single channels or the R, G, and B 3 channels. And has a four-dimensional structure in general.

2) Convolution Layer

An output feature map is formed by applying a specified number of 2D or 3D filters to image information from an input layer or a previous layer. Therefore, the number of output feature maps is equal to the number of specified filters. The convolutional layer computes the dot product of a vector between the weights (weights) of the filter and small patches of the image of the previous layer.

3) Activation Function

The activation function is ReLU (Rectified Linear Unit), which imparts nonlinearity to the neural network.

4) Pooling Layer

In pooling, an image is down-sampled through partial or regional maximum pooling or average pooling of the image.

5) Fully Connected Layer

Before the fully connected layer, flattening is performed on the last convolutional ray. The flattening layer is a layer that changes the data type of the CNN model into the form of FCNN (Fully Connected Neural Nestwork), and the result becomes the input of the fully connected layer. The fully connected layer computes the output score at the last layer. The size of the fully connected layer is 1x1xL, and L is the number of classes in the training data.

6) Classification function layer

When there are two or more classes in the classification, an output layer is formed using a classification function, for example, a Softmax function. The softmax function provides a method of predicting the discrete probability distribution for a class.

Training of the CNN model is performed by programming techniques for the model of the structure as described above, and in this process, the weight () update of the output layer is performed by backpropagation. As is well known, the backpropagation algorithm updates the weight of the output layer in proportion to the error, and then updates the weight of the hidden layer. When training for the above model is completed, a trained CNN model in the form of a target file is obtained.

The CNN model is used to determine the species of parrot in a mobile device according to the present invention. For this, the CNN model is converted into a model that can be used in mobile devices. The conversion of the model is performed by conversion of a file, and in this embodiment, it is converted into an open source file for Tensflolite. The mobile device modifies the relevant part (within about 10 lines) to use the model file for determining the parrot species, and builds the final APK file. This APK file is mounted on a mobile device, and this APK is loaded with a trained model to determine the species while the subject to be judged photographs the parrot.

Hereinafter, a detailed programming technique for collecting and expanding image data used in the experiment and generating a model using the same will be described.

In this experiment, 3 types of parrots were classified. A total of 2,535 images were obtained for each species of parrot, and 1,735 images (approximately 70%) as a training set, and a test set and a test set were obtained. Validation Set), divided into 400 sheets (about 15%)

Image data for training were largely obtained in two ways. The first is an image taken at Seoul Grand Park (see Fig. 4), and the second is an image obtained using Internet crawling by keyword search (see Fig. 5). About 500 images of each of the 20 parrot species possessed in Seoul Grand Park were secured. For each image, unnecessary background parts were removed and the image size was resized to a size of 640 x 480 pixels.

FIG. 4 is an example of a circular image obtained by separating in a frame unit from a video secured at Seoul National University Park, and FIG. 5 illustrates image data obtained by Internet crawling.

In the image data shown in Fig. 4, since a plurality of image files are secured in units of frames of a moving picture, these images are quite similar. So, if only such similar data is used, overfitting is likely to occur. To avoid this, different images of the same parrot species are needed.

Meanwhile, as a module for securing image data by crawling as shown in FIG. 5, "urllib", which is commonly used when data is fetched from the web, may be used. In the search structure where the URL of the search screen is made of the Get method, a URL that can query each species of parrot is constructed, and the binary image file is used as image data for training by requesting information about the page along with header information. You will be able to download it.

Image data was obtained by the method mentioned above, but more data is needed to sufficiently train the model. Therefore, image data can be further secured (expanded) by creating new data by slightly modifying existing data. To this end, the present invention expanded image data by transforming image data as follows.

1. Resale

The size is changed by multiplying the circular image data received as input by the specified value.

2. Wide-directional Shift

The circular image data is moved horizontally by the value of the specified range.

3. Height-directional shift

The circular image data is moved vertically within the specified range.

4. Zoom in or zoom out

The circular image is zoomed in or out by an arbitrary value within the specified range.

5. Flip

Invert the original image vertically or horizontally.

FIG. 6 illustrates a modified image in which a process of transforming the circular image in a frame unit of FIG. 4 is combined with the circular image. Referring to FIG. 6, left and right flip, zoom-in, and shift were performed on the circular image, and thus, a modified image of a shape different from that of the circular image was obtained. Such data transformation can also be performed on image data obtained by crawling as shown in FIG. 5. In addition, the expansion of the image data as described above can be performed for still images individually obtained, and through this, image data necessary for model training can be sufficiently secured.

Meanwhile, in the training of the model using the image data, Python3 (Python 3) was selected as the computer language for generating a model for predicting parrot species, and Keras (Keras) was selected for the prediction model generation. The reason for choosing Python3 is that there are rich libraries for data science and machine learning implemented, and since many experts actually use Python3, the community is greatly active. Tensorflow can be used to develop other predictive models. In the experiment of the present invention, Keras, which can be more intuitive and easily developed than tensorflow, was applied.

A module, for example, a module called tensorflow lite, can be used to load the model trained with the image data obtained by the above method into the app. However, in order to mount a model in an app through a specific module, for example, tensorflow lite, the size of each image needs to be changed because the size of the image is set to 224 x 224 pixels. The source below shows the flow of converting the file size of the input circular image to the size of 224 x 224.

<source code 1>

Meanwhile, it is necessary to create a model for training using the image resized as described above. This model can be created and trained using Keras as described above. In a class called Train of keras_train_and_test.py, functions for creating and training models are defined.

Specifically, set the keyword argument value of rescale=1./255 in the keras.preprocessing.image.ImageDataGenerator method of the Keras module. (Normalization of values related to 256 of RGB). Then, create a generator object by passing the necessary parameter (argument) to the flow_from_directory method. The parameters of this method include train, relative path value of validation folder, target size, class mode (class_mode), batch size, etc. In the above parameter, the class mode determines whether to perform binary classification or multi-classification. In the present invention, since it is a multi-classification, it is set as "categorical". On the other hand, the batch size represents the number of images brought to training at one time. For example, batch_size=10 indicates that 10 images are fetched per training. And, the model is created in the set_model method of the Train class object.

The following describes how to create a model using keras.Sequential.

Code 2 below exemplifies a method of generating a Sequential model in which the layers are stacked (added) by passing a list of multiple layer instances to a constructor.

<source code 2>

Source code 3 below exemplifies how to stack layers on the stack by the add method after model creation.

<source code 3>

In Keras' model object, parameter is passed to method ADD as model.add(keras.layers.Conv2D(16, kernel_size=(3, 3), activation="relu", input_shape=(224, 224, 3))) If so, it adds a convolutional 2D layer, at this time, there are 16 filters (kernels) of size 3X3, and relu is used as the activation function, and an image of size 224 x 224 is used, and the channel of this image is It is designated as 3(R, G, B). You can stack MaxPool2D, Dropout, and Flatten radars in the same way.

After the training model is generated by the above method, training of the model is performed using the training image data. This training can be started in the "start_train" method, which includes, for example, the "model.compile" method and the "self.model.fit_generator" method.

The parameter loss, optimizer, and metrics of the model's compile method (self.model.compile) can be defined as follows.

<Source code 4>

1.loss = ’categorical_crossentropy’

This means that the loss value (=cost) is measured as "crossentropy" for multiple classes.

2. optimizer=’adam’

This means that a technique called adam is used to reduce the loss value and adjust the weights values to reach the local maximum.

3. metrics=[‘accuracy’]

This means that we will evaluate based on accuracy. The criteria for evaluating whether or not it is a good model include accuracy, precision, recall, and f1-score.

<Source Code 5>

Training starts in earnest by calling the method "self.model.fit_generator" after calling the compile method of source code 4. self.train_generator is an image-related generator object. Looking at the parameters of the method above, it trains 24 times (steps_per_epoch=24) per 1 epoch, and performs a total of 20 epochs (epochs=20). 1 epoch means that the weight of the model is adjusted using all of the train data.

validation_data=self.validation_generator is used to prevent the model from overfitting to the training data set (a tendency to stick to the features of the training data rather than catching features of the parrot). Parameter validation_steps=27 means that 27 times the process of preventing overfitting by using the validation data.

You can evaluate whether the model created through the Test class actually works well.

Like when creating Train class, create Test class using keras.preprocessing.image.ImageDataGenerator.

<Source Code 6>

keras.preprocessing.image.ImageDataGenerator(rescale=1./255), flow_from_directory(".\\pic\\test", target_size=(224, 224), class_mode="categorical", batch_size=5)

The Test class is a generator object related to test image data. The evaluation process can be executed by the following method.

<Source Code 7>

model.evaluate_generator(self.test_generator, steps=27)

In the above source code 7, steps=27 means that the evaluation process will be performed 27 times.

Meanwhile, the evaluation result can be printed by the following print function.

<Source Code 8>

print("%s: %.2f%%"%(self.model.metrics_names[1], score[1]*100))

Print out what% accuracy is through the print function above. In print_classfication_report, evaluation items related to confusion_matrix are output.

The model after training is saved as a file. For example, Keras can be saved as follows.

<Source Code 9>

file_name = "trained_model.h5"

keras.models.save_model(self.model, file_name)

The above source code saves the trained model in the current working directory under the file name "trained_model.h5".

As described above, it is necessary to change the file of a specific module, for example, a model trained by Keras, to be usable in a mobile device, for example, to be ported and used in TensFlow Lite.

The Converter class of convert_from_keras_to_tflite.py has a function to convert the saved h5 file into a tflite file for .Tensflow Lite.

<Source Code 10>

converter =

lite.TFLiteConverter.from_keras_model_file("trained_model.h5")

tfilite_mode = conveter.comver()

The above source code 10 creates a "converter" object that converts "trained_model.h5" created above to "trained_model..tflite", and in converter.convert() it actually converts and saves it as tfilite_mode.

<Source Code 11>

open("trained_model.tflite", "wb").write(tflite_model)

The open function above opens a file called trained_model.tflite in binary write mode and saves it as tflte_model data. This will finally create a file called trained_model.tflite. This file is stored together with the parrot species determination app in a parrot type determination device using this, for example, a portable terminal device. For example, if the trained_model.tflite file is inserted into the assets folder of the Android Studio project of the app to be used in TensFlow Lite, and the project is built by connecting a portable device to be used as a parrot type determination device, the parrot species is determined. The app is installed on the device.

<Source Code 12>

<Source Code 13>

Source codes 12 and 13 above show the code modification part of the modified Tensflolite project. Here, put the tflite file created earlier and the text file (labels_test.txt) with the label written on it into the asset folder and test. Create a class for classifiers for tflite files. Then we change the part of the code that connects the camera activity and the fragment.

7 to 9 show a result of recognizing a parrot to be determined by a camera by a portable terminal device after the app is built. Figure 7 is a species of parrot is judged as ara chloroptera with a probability of 94%, Figure 8 is Cacatua goffiniana (white-fronted sulfur parrot) is judged with a probability of about 92% as 0.92, Figure 9 Is Eclectus roratus (New Guinea parrot), which judges parrot species with a probability of 1.0, that is, 100%, and as a result of the actual final determination, it was confirmed that the species was judged 100% accurately.

The portable terminal device uses a built-in camera to capture a parrot to be determined in real time, and applies the captured image information to the trained model to determine the species of the parrot probabilistically. Pixel information (pixel values) of 3 channels R, G, and B of the frame of the parrot image acquired through the camera of the mobile terminal device is processed through a convolutional neural network (CNN) process, and is then processed through the final Softmax algorithm to a specific species. A vector representing the probability for is obtained. To this end, the pixel information of the image is multiplied by a fixed weight value of a model that has been trained in advance, and when these values are added, an output value in the form of a matrix is finally obtained. This output value is flattened to form a fully connected layer, from which the number of labels (classification) desired through a multi-classification activation function, e.g., softmax algorithm (in this example, 3 parrots, so factor 3 A vector with elements). Among the values of this vector, the maximum value becomes the final predicted value that is the criterion for determining a specific species, and the vector value and the label of the corresponding species are output to the portable terminal device. For example, if the vector value is [0.3, 0.6, 0.1], the predicted value is parrot number 2 (label 2). This is because the probability of being parrot 2 is 0.6, which is the highest among the three.

In the description of the above embodiment, classification was attempted only on three types of parrots, but the number of classified species may be expanded through training on more diverse species.

In the present invention, the lack of training data expands the transformed data, and through this, it is possible to construct a trained model in more various forms. In the present invention, in determining the parrot species, real-time determination is possible through a portable terminal device, which enables rapid and accurate processing. The present invention makes it easy to obtain information on the species of each individual about smuggled pets, especially parrots, which are required to be applied not only in stores but also in customs clearance.

As shown in Fig. 10, the parrot species determination apparatus according to the present disclosure includes: a memory storing a final model file (trained model) obtained by the method; A camera for obtaining comparison image data from the parrot to be determined; By loading an end determination app (APK), a fully connected layer is formed through a convolutional neural network technique using the trained weight on the comparative image data acquired by the imaging device, and multi-classification is activated for the fully connected layer. A processor that forms an output layer (output vector) containing the parrot information using a function; It may include a; display for outputting the parrot information by the processor.

The processor is a CPU or GPU, and the present invention is not technically limited by a specific processor.

In the parrot species determination apparatus, the processor may apply a softmax algorithm as a multi-classification activation function.

In addition, the processor, such as a CPU or GPU, may change the width or height of the type image, enlarge or reduce it, or flip the deformation data in the horizontal or vertical direction in forming type deformation data from the type image data. Can be formed.

As described above, exemplary embodiments of the present invention have been described in detail, but those of ordinary skill in the art to which the present invention pertains, without departing from the spirit and scope of the present invention defined in the appended claims. It will be possible to practice the present invention by various modifications. Therefore, changes to the embodiments of the present invention will not be able to depart from the technology of the present invention.

Claims (10)

Collecting a plurality of raw images for each type of parrot;
Forming a modified image for each type by transforming the collected circular image for each type into another shape;
Preparing training data by collecting the original image and the transformed image;
Generating a model having a multidimensional convolutional neural network structure, training the model using the training data in which the circular image data and the transformed image are collected, and generating a model file including weights trained for each parrot species;
Converting the model file into a final model file for a portable terminal device and mounting it on the portable terminal device;
Obtaining a comparison image from a parrot to be determined using a photographing device provided in the portable terminal device;
Forming a fully connected layer through a convolutional neural network technique using the trained weight on the comparison image; And
Forming an output layer containing probability information of the parrot using a multi-classification activation function for the fully connected layer; Parrot species determination method comprising a. The method of claim 1,
A parrot species determination method to which a softmax algorithm is applied as a multiple classification activation function. The method according to claim 1 or 2,
The step of forming a deformed image from the circular image comprises forming a deformed image through a change in the width or height of the circular image, parrot species determination method. The method according to claim 1 or 2,
The step of forming a deformed image from the circular image includes forming a deformed image through enlargement or reduction of the circular image. The method according to claim 1 or 2,
The forming of the deformation image from the circular image comprises forming a deformation image through horizontal or vertical flipping of the circular image. A memory for storing the final model file obtained by the method according to claim 1 or 2;
A photographing device for obtaining a comparison image from the parrot to be determined;
A complete connection layer is formed through a convolutional neural network technique using the trained weight for the comparison image acquired by the photographing device, and an output layer containing the parrot information is generated using a multi-classification activation function for the fully connected layer. Forming processor; And
Parrot species determination apparatus comprising a; a display for outputting information on the parrot by the processor. delete delete delete delete

Images