KR102504607B1 - Multidimensional visualization system using deep learning object detection ratio and filtering technology and the method using the same - Google Patents

Abstract

The present invention relates to an image visualization system and method using deep learning-based object recognition that can detect multiple objects using image object recognition, introduce detection filtering technology accordingly, and then solve and visualize spaced clustering. a) transmitting the input image to the server for classification, b) detecting objects included in the image in the server, c) bounding boxes of the objects detected in step b) extracting information; d) obtaining the area of the bounding box for each object extracted in step c) and calculating the area ratio between the bounding box and the image; e) the area ratio calculated in step d) Selecting the largest object and extracting a feature map from the selected object through a convolutional neural network, f) Clustering the feature maps extracted in step e) through T-SNE (T-Stochastic Neighbor Embedding) algorithm and g) visualizing the image coordinates clustered in step f).

Description

Multidimensional visualization system using deep learning object detection ratio and filtering technology and the method using the same}

The present invention relates to a multi-dimensional visualization system and method to which multi-object detection ratio and filtering technology using deep learning-based object recognition are applied. It relates to a multidimensional image visualization system and method using deep learning-based object recognition, which is characterized by extracting and filtering the features of a degraded area and then applying it to a probabilistic distribution algorithm to visualize.

Recently, the way of expressing and accepting data is rapidly changing, but in the case of most general visualization solutions or application models, visualization services are provided using only chart and text-based information.

Accordingly, the necessity of providing a service model to which a new expression method is applied is emerging based on an image analysis technique, which is actively researched among unstructured data.

Therefore, it is time to identify the need for automatic classification and filtering technology for various visualizations according to image classification, and to develop a new concept multidimensional space visualization service model by utilizing probability distribution and correlation.

For automatic classification of images, various deep learning object recognition technologies should be used to create a model that classifies each object from image data, extract features of the data, and express them as vectors.

In addition, in order to visualize and express the extracted high-dimensional vector information, a method using a probabilistic distribution algorithm capable of clustering by projecting each feature to a low-dimensional is required.

“Research on image-based visualization of real-time numerical data” (Eunhee Cho, Hyunwook Kim, Hanyoung Ryu, Digital Design Studies Vol. 11, No. 4, 2011.10.)

The present invention has been made to solve the above problems, and an object of the present invention is to provide a new visualization service through dimensionality reduction of high-dimensional data.

In addition, the present invention detects multiple objects by utilizing deep learning object recognition in image data, creates a bounding box for each detected object, extracts and filters the characteristics of the selected object from it, and then applies it to a probabilistic distribution algorithm. It aims to apply and visualize.

In order to solve the above problems, the present invention is a) transmitting an input image to a server for classification, b) detecting an object included in the image in the server, c) the b) step extracting bounding box information of the objects detected in step d) obtaining the area of the bounding box for each object extracted in step c) and calculating the area ratio between the bounding box and the image; e ) Selecting an object having the largest area ratio calculated in step d) and extracting a feature map from the selected object through a convolutional neural network, f) T-SNE (T -Stochastic Neighbor Embedding) algorithm, and g) visualizing the clustered image coordinates in step f).

In addition, the bounding box of step c) may be a rectangle surrounding the object detected in step b).

Further, in the step c), the coordinates of the outermost vertices of the bounding box are obtained to obtain the horizontal and vertical lengths of the bounding box, and in the step d), the horizontal and vertical lengths of the bounding box obtained in step c) are obtained. It is characterized in that the area of each bounding box is calculated using

In addition, the step e) is characterized in that a feature map is extracted by passing the selected object through a VGG-16 network.

As described above, the image visualization system and method using deep learning-based object recognition according to various embodiments of the present invention can improve clustering performance by learning data from which unnecessary features are removed from a large amount of multidimensional data. It has the effect of visualizing the information presented.

In addition, according to the present invention, by utilizing the image filtering method according to object detection, it can be used in the medical industry market that requires precise analysis of microorganisms, cells, etc. By bringing it, it is expected to access not only e-commerce, but also the online education market or the cultural market combined with VR.

In addition, the present invention develops a service model by applying a multi-touch script to actively respond to offline and mobile markets, so that promotion in the signage market using DID multi-vision and market preoccupation can be expected.

1 is a flowchart of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention;
2 is a schematic diagram of step a) of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention;
3 is a schematic diagram of Faster R-CNN performed in step b) of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention;
4 is a schematic diagram of step b) of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention;
5 is a schematic diagram of steps c) and d) of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention;
6 is a schematic diagram of a VGG-16 algorithm performed in step e) of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention;
7 is a block diagram of an image visualization system using deep learning-based object recognition according to an embodiment of the present invention.

Hereinafter, an image visualization method using deep learning-based object recognition according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart of an image visualization classification method using deep learning-based object recognition according to an embodiment of the present invention.

As shown in FIG. 1, the image visualization method using deep learning-based object recognition according to an embodiment of the present invention is sequentially performed in steps a), b), c), d), Step e) and step f) may be included.

In the present invention, a) transmitting an input image to a server for classification, b) detecting an object included in the image in the server, c) a bounding box of the objects detected in step b) ) extracting information, d) obtaining the area of the bounding box for each object extracted in step c), and calculating the area ratio between the bounding box and the image, e) the area ratio calculated in step d) Selecting the largest object and extracting a feature map from the selected object through a convolutional neural network; and g) visualizing the image coordinates clustered in step f).

2 is a schematic diagram of step a) of an image visualization method using deep learning-based object recognition according to an embodiment of the present invention.

Step a) transmits the input image to the server 100 for classification. Step a) may be manually performed by a web administrator, or when users upload an image to a specific space, the management device automatically transmits the uploaded image to the server 100.

Step b) detects an object included in the image transmitted to the server 100, and step c) extracts bounding box information of the objects detected in step b). Steps b) and c) may be performed sequentially or concurrently.

An object included in an image means a meaningful part among things included in the image. In step b), an object included in the image may be recognized through a Faster Regions-Convolutional Neural Networks (R-CNN) neural network.

3 is a schematic diagram of Faster R-CNN. Faster R-CNN is a deep learning model for learning multiple objects in one image. Region Proposal generation is replaced with a new method and integrated into the model to end- It is an artificial neural network constructed in a to-end manner.

4 illustrates an example of an image uploaded in step a) to explain steps b) and c).

4 shows a dog taking a walk and a man taking the dog for a walk. Here, the objects may be a dog and a man, and the dog is referred to as a first object A1 and the man referred to as a second object A2.

Step b) detects the first object A1 and the second object A2 shown in FIG. 4 in the same way as deep learning or machine learning such as Faster R-CNN.

The bounding box extracted in step c) refers to the smallest border line surrounding an object. In this embodiment, the bounding box may have a non-inclined rectangular shape.

Step c) obtains location information of the outermost pixels of each of the second objects A2 of the first object A1, that is, location information of the vertex of each object's bounding box. That is, in the step c), the coordinates of the outermost vertices of the bounding box are obtained, and the horizontal and vertical lengths of the bounding box are obtained (FIG. 5).

In step d), the area of the bounding box for each object detected in step c) is obtained, and an area ratio representing the ratio between the area of the bounding box and the area of the entire image is calculated.

That is, in step d), the area of each bounding box is calculated using the horizontal and vertical lengths of the bounding boxes obtained in step c), and the bounding boxes for each image, first object A1, and second object A2 are calculated. By calculating the area ratio of , Oir (Object image ratio) is obtained.

Oir(Object image ratio) = area of bounding box for each object / total image area

In step e), an object having the largest area ratio calculated in step d) may be selected or an object having an area ratio greater than or equal to a reference value may be selected, and a feature map may be extracted from the selected object through a convolutional neural network.

Step e) may select an object whose area ratio calculated in step d) is greater than or equal to a reference value, where the reference value may vary as needed or may be preset.

In this case, if there are several objects whose area ratio is greater than or equal to the reference value, the objects may be selected by assigning weights in order of area ratio.

In the present embodiment shown in FIG. 5, in step e), only the second object A2 with a large area ratio can be selected, excluding the first object A1 with a small area ratio.

Step e) extracts a feature map of the selected object through a convolutional neural network.

In step e) of the present embodiment, a feature map of the selected object may be extracted using a pre-trained VGG-16 Convolutional Neural Networks (VGG-16) neural network.

6 shows a schematic scheme of the VGG-16 algorithm.

Referring briefly to the VGG-16 algorithm with reference to FIG. 6, the VGG-16 algorithm may first receive a 224×224×3 image. The last 3 of the image specification means RGB. Afterwards, the VGG-16 algorithm convolves the input image with 64 3×3×3 filter kernels in the first layer, and stride and zero padding can be set separately. In the first layer, 64 224×224 feature maps are created as a result, and the ReLU function is applied to activate them. The ReLU function is always applied except for the last 16 layers, and after the application of the VGG-16 algorithm, it consists of 1,000 neurons, and the output values are activated with the softmax function. Being composed of 1,000 neurons means that the network was created for the purpose of classifying into 1,000 classes.

In step f), the feature maps for each object extracted in step e) are image clustered through T-SNE (T-Stochastic Neighbor Embedding) algorithm. T-SNE is one of the machine learning algorithms used for dimensionality reduction of data. T-SNE is a non-linear dimensionality reduction technique, which is useful for visualizing high-dimensional data by reducing it to 2D or 3D.

T-SNE transforms the distance between data into stochastic probaility and then embeds it to learn a 2-dimensional embedding vector that preserves the neighbor structure between data expressed as a high-dimensional vector. It is a stochastic distribution algorithm. That is, the distance between each multidimensional data can be probabilistically calculated and clustered.

In step g), the image coordinates clustered in step f) may be visualized and provided to the user. In step g), clustered image coordinates can be visualized on a web browser by utilizing a WebGL component such as Three.js.

7 is a block diagram of an image visualization system using deep learning-based object recognition according to an embodiment of the present invention.

As shown in FIG. 7, the image visualization system using deep learning-based object recognition according to an embodiment of the present invention includes a web browser 10, a management server 20, an image analysis server 30, an image An analysis result DB 31 and an image result server 40 may be included.

In the web browser 10, an administrator or user uploads an image or analyzes and manages the image. Specifically, after step a) is performed or steps a) to f) of the present invention described above are performed, step g) is executed on the web browser 10 to visualize and output the result of the T-SNE algorithm. can

The management server 20 checks the license of an administrator or user who is connected, manages the addition, deletion, compression, and transmission of files, and transmits a signal to the image analysis server 30 to be described later to perform the T-SNE algorithm. play a role The management server 20 may be linked with the framework DB 21, and operation and log information of the management server 20 may be stored in the framework DB 21.

The image analysis server 30 receives an image file uploaded by an administrator or a user from the management server 20 and becomes a subject that performs steps b) to f). That is, when an image is received from the management server 20, steps before the T-SNE algorithm is applied, such as object detection, bounding box detection, area ratio calculation, VGG-16 algorithm execution, and feature map extraction, are performed on the image, and then Step f) may be performed through the T-SNE algorithm. The work result of the image analysis server 30 may be stored in the image analysis result DB 31 and may be transmitted to the image result server 40 .

The image result server 40 serves to output so that the user can access and check the image analysis result.

A firewall and security network may be installed between the web browser 10, the management server 20, the image analysis server 30, the image result server 40, and the user terminal 50.

The present invention is not limited to the above embodiments, and the scope of application is diverse, and various modifications and implementations are possible without departing from the gist of the present invention claimed in the claims.

10: Web browser
20: management server
21 : Framework DB
30: image analysis server
31: Image analysis result DB
40: Image result server
50: user terminal
100: server
A1: first object
A2: Second object

Claims (5)

a) transmitting an input image to a server to be classified in a web browser;
b) detecting an object included in the image by the server;
c) extracting bounding box information of the objects detected in the step b) in the server;
d) obtaining an area of a bounding box for each object extracted in step c) in the server, and calculating an area ratio between the bounding box and the image;
e) selecting an object having the largest area ratio calculated in step d) or an object having an area ratio greater than or equal to a reference value, and extracting a feature map from the selected object through a convolutional neural network;
f) clustering images from the feature maps extracted in step e) in the server through a T-Stochastic Neighbor Embedding (T-SNE) algorithm; and
In the image visualization method using deep learning-based object recognition, including g) visualizing the image coordinates clustered in step f) in the web browser,
Step b) recognizes an object included in the image through a Faster Regions-Convolutional Neural Networks (R-CNN) neural network,
In the step c), the coordinates of the outermost vertices of the bounding box are obtained, and the horizontal and vertical lengths of the bounding box are obtained;
Step d) calculates the area of each bounding box using the horizontal and vertical lengths of the bounding boxes obtained in step c);
In the step e), if there are several objects whose area ratios are greater than the reference value, weights are given in order of area ratios to select objects,
Step f) is an image visualization method using deep learning-based object recognition, characterized in that for clustering by probabilistically calculating the distance between high-dimensional data.
According to claim 1,
The image visualization method using deep learning-based object recognition, characterized in that the bounding box of step c) is a rectangle surrounding the object detected in step b).
delete According to claim 1,
Step e) is an image visualization method using deep learning-based object recognition, characterized in that for extracting a feature map by passing the selected object through a VGG-16 network.
delete

Images