CN111832588A - Riot and terrorist image labeling method based on integrated classification - Google Patents
Riot and terrorist image labeling method based on integrated classification Download PDFInfo
- Publication number
- CN111832588A CN111832588A CN201910315804.3A CN201910315804A CN111832588A CN 111832588 A CN111832588 A CN 111832588A CN 201910315804 A CN201910315804 A CN 201910315804A CN 111832588 A CN111832588 A CN 111832588A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- sub
- label
- labeling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
Abstract
The invention discloses an riot and terrorist image labeling method based on integrated classification. The method comprises the following steps: normalizing the image to be annotated to 224 multiplied by 224 and then inputting the image annotation integrated network; each sub-network in the integrated network maps the extracted image features into a label semantic space to obtain a label probability vector with the size of 1 multiplied by N of an image to be labeled; combining the label probability vectors output by each sub-network into a matrix, and performing a series of operations with the weight distribution matrix to obtain a label probability vector with the final size of 1 xN of the image to be labeled; and setting a threshold value for the calculated label probability vector, wherein all labels larger than the threshold value are the final labeling result of the image to be labeled. The labeling method described by the invention has the advantages of short network training time, high labeling accuracy, strong stability and the like, and compared with the traditional machine learning, the labeling method has great improvement on the labeling accuracy and the recall rate, and has certain practical value in the specific field of terrorist-related information.
Description
Technical Field
The invention designs an ensemble classification-based violence and terrorist image labeling method, and relates to the technical field of deep learning and computer vision.
Background
With the rapid development of internet social platforms and the popularization of image acquisition devices such as mobile phones and digital cameras, image and video data that people can access everyday show explosive growth. The mass image data brings convenience to daily life of people, and meanwhile, the partial violence images bring negative effects to the health growth of society and harmonious teenagers. How to effectively manage these data becomes a problem that needs to be solved urgently. The automatic image annotation technology gradually becomes one of the key technologies in the fields of image analysis and application due to the characteristic that text characteristic information reflecting the content of the image is automatically added to the image.
The current automatic image labeling methods are mainly divided into two categories: an image labeling method based on classification idea and an image labeling method based on a correlation model. The image labeling method based on the classification idea treats the automatic labeling of the image as an image classification problem, treats each labeling keyword as a category, firstly divides the image into regular areas, and then classifies the divided image areas. The image labeling method based on the correlation model comprises the steps of firstly extracting visual features such as color and texture of an image or an image area, then calculating joint probability distribution between the visual features of the image and labeling keywords, and finally labeling the image or the area to be labeled by establishing a probability correlation model.
The traditional method makes a certain progress in the field of image annotation, but because characteristics need to be manually selected, information loss is caused, so that the annotation precision is insufficient, and the recall rate is low; although the deep learning model achieves higher achievement in the field of image recognition and classification, most of the deep learning model aims at improvement of network or single-label learning, and application and improvement of image labeling belonging to multi-label learning are less. Imbalances in label classes in multi-label databases can also lead to a reduction in the quality of the trained model labeling.
Disclosure of Invention
The invention provides an integration classification-based riot and terrorist image labeling method, which treats the labeling problem of the riot and terrorist image as a multi-label classification problem according to a classification thought, and improves the labeling accuracy of each type of label by a model through the improvement of an integration network combination module in consideration of the condition of uneven distribution of labeling key words.
The invention realizes the purpose through the following technical scheme, which comprises the following steps:
the method comprises the following steps: and after the image to be annotated is normalized to 224 multiplied by 224, the image to be annotated is input into the riot and terrorist image annotation integrated network, and the image characteristics extracted by each sub-network in the integrated network are obtained.
Step two: and each sub-network maps the extracted image features to a label semantic space to obtain a label probability vector of the image to be labeled, wherein the size of the label probability vector obtained by each sub-network is 1 multiplied by N.
Step three: and combining the label probability vectors output by each sub-network into a matrix, and performing a series of operations on the matrix and the weight distribution matrix to obtain a final label probability vector of the image to be labeled, wherein the size of the final label probability vector is 1 multiplied by N.
Step four: and setting a threshold value for the label probability vector obtained by calculation in the step three, wherein all labels larger than the threshold value are the final labeling result of the image to be labeled.
Step one, the riot and terrorist image labeling integrated network consists of a plurality of sub-networks, and the training steps of the sub-networks are as follows:
(1) the method comprises the steps of manufacturing a data set, wherein the data set is an image containing a plurality of riot and terror elements, the corresponding label is an N-dimensional vector, and for each type of riot and terror elements, if the image contains the type of riot and terror elements, the corresponding dimension of the label vector is 1, otherwise, the label vector is 0; due to the problem of uneven label distribution of the data set, the invention adopts replaced random sampling on each type of label-containing images in sample processing, and combines the samples into a data-balanced sampling set for training the sub-network.
(2) The network structure of the sub-network is a convolutional neural network, all parameters obtained by training the sub-network in an ImageNet classification task are applied to a riot and terrorist image labeling task through transfer learning, and a full connection layer is additionally added at the last of the sub-network for training.
The weight distribution matrix in the third step is essentially that after the training of the sub-networks is completed, the sub-networks are subjected to weight distribution according to different classification accuracy rates of the sub-networks on the same label class, and then the weight distribution matrix W is obtainedN×MCan be represented by the following formula:
WN×M＝[W1,W2,...,WN]T
wherein Wi＝[wi1,wi2,...,wiM]Representing the weight distribution of the i-th class label on the M sub-networks, and enabling Oi＝[oi1,oi2,...,oiM]The prediction probability values of the ith class label representing the input image on M sub-networks, and the final label probability vector Prob1×NFrom WiAnd OiThe product of (a) is calculated to yield:
Prob1×N＝[W1·O1 T,W2·O2 T,...,WN·ON T]。
the threshold value in the fourth step is usually set to be an empirical value of 0.5, and a relatively good image labeling effect can be achieved through testing the threshold value.
The invention mainly aims to provide two improvements:
(1) the improvement of the training mode of the sub-network is different from the common training mode of transfer learning, namely the output dimension of the last full connection layer of the neural network is changed into the classification dimension of a target task for training, all parameter information of a pre-training model is reserved, and a full connection layer is additionally added at the last of the sub-network for training.
(2) The improvement of the sub-network combination mode is a network fusion mode for carrying out weight distribution on each sub-network according to different classification accuracy rates of the same label class of each sub-network, the improvement mode solves the problem of unbalanced label class in a data set, and the labeling accuracy rate of the integrated network to each class of labels is improved.
Drawings
FIG. 1 is a schematic diagram of the labeling result of an image of a riot and terrorist
FIG. 2 is a flowchart of an ensemble classification-based violence and terrorist image labeling method according to the present invention
FIG. 3 is a flow chart of the present invention for training subnetworks based on transfer learning
Detailed Description
The invention is further described below with reference to the accompanying drawings:
fig. 1 is a schematic diagram of an image annotation result of a riot terrorism. The riot and terrorist image labeling network can label the riot and terrorist elements such as guns, ships, airplanes, fires, armored vehicles, artillery and the like in the image.
In fig. 2, a riot and terrorist image labeling method based on ensemble classification includes the following steps:
the method comprises the following steps: and respectively taking replaced random samples of each type of label containing images in the initial training set, and combining the samples into a data balanced sample set for training each sub-network (namely, the individual learners in the graph) of the integrated network.
Step two: parameters obtained by training the convolutional neural network in the ImageNet classification task are applied to the violence image labeling task through transfer learning to generate mutually independent sub-networks.
Step three: and combining the output results of the sub-networks in a weight distribution mode to obtain a final labeling result.
FIG. 3 is a flow chart of the present invention for training subnetworks based on migration learning. The common transfer learning training method is to change the output of the last full connection layer of the neural network from 1000 dimensions to the category number of a target task for training, in the violence and terrorism image labeling task, in order to keep the pre-training information of the full connection layer, the pre-training information of the convolutional neural network is completely transferred to the task, and the full connection layer with the input of 1000 dimensions and the output of label dimensions is added at the end of the network for training, so that the training mode saves the training time and improves the labeling accuracy of the sub-network.
In order to verify the effectiveness of the ensemble classification-based violence and terrorist image labeling method, the method provided by the invention is compared with a classical machine learning algorithm SVM, a decision tree C4.5 and KNN, and the effectiveness of each method is evaluated according to three indexes of average accuracy P, average recall rate R and F1 values of the labeling result, so that the experimental result of each image labeling method shown in the table 1 is obtained.
TABLE 1 Experimental results of the labeling methods for each image
Experiments show that the SVM performs best in the traditional machine learning method, and compared with the SVM, the method provided by the invention has the advantages that the accuracy is improved by 26%, the recall rate is improved by 30%, and the improvement is great.
In order to verify the improved effectiveness of the method in the combination mode, the weight distribution combination mode proposed by the method is compared with the averaging and majority voting combination mode commonly used in ensemble learning, and the accuracy of different combination modes on each label category is shown in table 2.
TABLE 2 comparison of accuracy of different combinations on each label class
As can be seen from the table above, the labeling accuracy of the weight distribution combination mode on most label categories is highest, and the numerical value fluctuation is small. The explanation shows that the labeling network using the weight distribution combination mode is stable on different label types, and the situation that the difference of accuracy on each label type is large due to the problem of uneven distribution of the labels of the data set does not exist.
Claims (5)
1. An violence and terrorism image labeling method based on integrated classification is characterized by comprising the following steps:
the method comprises the following steps: normalizing the image to be annotated to 224 multiplied by 224 and then inputting the image to be annotated into the riot and terrorist annotation integrated network to obtain the image characteristics extracted by each sub network in the integrated network;
step two: each sub-network maps the extracted image features to a label semantic space to obtain a label probability vector of the image to be labeled, and the size of the label probability vector obtained by each sub-network is 1 multiplied by N;
step three: combining the label probability vectors output by each sub-network into a matrix, and performing a series of operations with the weight distribution matrix to obtain a final label probability vector of the image to be labeled, wherein the size of the final label probability vector is 1 multiplied by N;
step four: and setting a threshold value for the label probability vector obtained by calculation in the step three, wherein all labels larger than the threshold value are the final labeling result of the image to be labeled.
2. The riot terrorist image annotation integrated network of claim 1, comprising a plurality of sub-networks, wherein the sub-networks are trained by the following steps:
(1) the method comprises the steps of manufacturing a data set, wherein the data set is an image containing a plurality of riot and terror elements, the corresponding label is an N-dimensional vector, and for each type of riot and terror elements, if the image contains the type of riot and terror elements, the corresponding dimension of the label vector is 1, otherwise, the label vector is 0;
(2) the network structure of the sub-network is a convolutional neural network, parameters obtained by training the sub-network in the ImageNet classification task are applied to the violence image labeling task through transfer learning, the training speed of the sub-network can be increased, and a loss function used for guiding the training of the sub-network is cross entropy loss.
3. The weight distribution matrix of claim 1, wherein the weight distribution matrix is substantially configured to distribute the weights of the sub-networks according to different classification accuracy rates of the sub-networks for the same label class after the sub-networks are trained, so as to further improve the accuracy of the labeling result.
4. The threshold value of claim 1 is usually set to an empirical value of 0.5, and the threshold value is tested to achieve a better image labeling effect.
5. N as claimed in claim 1 and claim 2 is an integer greater than 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910315804.3A CN111832588A (en) | 2019-04-18 | 2019-04-18 | Riot and terrorist image labeling method based on integrated classification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910315804.3A CN111832588A (en) | 2019-04-18 | 2019-04-18 | Riot and terrorist image labeling method based on integrated classification |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111832588A true CN111832588A (en) | 2020-10-27 |
Family
ID=72915038
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910315804.3A Pending CN111832588A (en) | 2019-04-18 | 2019-04-18 | Riot and terrorist image labeling method based on integrated classification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111832588A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112801237A (en) * | 2021-04-15 | 2021-05-14 | 北京远鉴信息技术有限公司 | Training method and device for violence and terrorism content recognition model and readable storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101281521A (en) * | 2007-04-05 | 2008-10-08 | 中国科学院自动化研究所 | Method and system for filtering sensitive web page based on multiple classifier amalgamation |
| CN102298606A (en) * | 2011-06-01 | 2011-12-28 | 清华大学 | Random walking image automatic annotation method and device based on label graph model |
| CN104166548A (en) * | 2014-08-08 | 2014-11-26 | 同济大学 | Deep learning method based on motor imagery electroencephalogram data |
| EP2991003A2 (en) * | 2014-08-28 | 2016-03-02 | Baidu Online Network Technology (Beijing) Co., Ltd | Method and apparatus for classification |
| CN108023876A (en) * | 2017-11-20 | 2018-05-11 | 西安电子科技大学 | Intrusion detection method and intruding detection system based on sustainability integrated study |
| CN108111614A (en) * | 2017-12-27 | 2018-06-01 | 北京工业大学 | Agriculture and forestry pest and disease monitoring manages system |
| CN108416384A (en) * | 2018-03-05 | 2018-08-17 | 苏州大学 | A kind of image tag mask method, system, equipment and readable storage medium storing program for executing |
| CN109376773A (en) * | 2018-09-30 | 2019-02-22 | 福州大学 | Crack detecting method based on deep learning |
| CN109471942A (en) * | 2018-11-07 | 2019-03-15 | 合肥工业大学 | Chinese comment sensibility classification method and device based on evidential reasoning rule |
| CN109492105A (en) * | 2018-11-10 | 2019-03-19 | 上海文军信息技术有限公司 | A kind of text sentiment classification method based on multiple features integrated study |
-
2019
- 2019-04-18 CN CN201910315804.3A patent/CN111832588A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101281521A (en) * | 2007-04-05 | 2008-10-08 | 中国科学院自动化研究所 | Method and system for filtering sensitive web page based on multiple classifier amalgamation |
| CN102298606A (en) * | 2011-06-01 | 2011-12-28 | 清华大学 | Random walking image automatic annotation method and device based on label graph model |
| CN104166548A (en) * | 2014-08-08 | 2014-11-26 | 同济大学 | Deep learning method based on motor imagery electroencephalogram data |
| EP2991003A2 (en) * | 2014-08-28 | 2016-03-02 | Baidu Online Network Technology (Beijing) Co., Ltd | Method and apparatus for classification |
| CN108023876A (en) * | 2017-11-20 | 2018-05-11 | 西安电子科技大学 | Intrusion detection method and intruding detection system based on sustainability integrated study |
| CN108111614A (en) * | 2017-12-27 | 2018-06-01 | 北京工业大学 | Agriculture and forestry pest and disease monitoring manages system |
| CN108416384A (en) * | 2018-03-05 | 2018-08-17 | 苏州大学 | A kind of image tag mask method, system, equipment and readable storage medium storing program for executing |
| CN109376773A (en) * | 2018-09-30 | 2019-02-22 | 福州大学 | Crack detecting method based on deep learning |
| CN109471942A (en) * | 2018-11-07 | 2019-03-15 | 合肥工业大学 | Chinese comment sensibility classification method and device based on evidential reasoning rule |
| CN109492105A (en) * | 2018-11-10 | 2019-03-19 | 上海文军信息技术有限公司 | A kind of text sentiment classification method based on multiple features integrated study |
Non-Patent Citations (2)
| Title |
|---|
| 严寒: "基于卷积神经网络的图像分类算法研究", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
| 李艳秋: "基于集成学习的人脸识别研究", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112801237A (en) * | 2021-04-15 | 2021-05-14 | 北京远鉴信息技术有限公司 | Training method and device for violence and terrorism content recognition model and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112308158B (en) | Multi-source field self-adaptive model and method based on partial feature alignment | |
| WO2021042828A1 (en) | Neural network model compression method and apparatus, and storage medium and chip | |
| CN110162593B (en) | Search result processing and similarity model training method and device | |
| CN111881714B (en) | Unsupervised cross-domain pedestrian re-identification method | |
| CN111126386B (en) | Sequence domain adaptation method based on countermeasure learning in scene text recognition | |
| CN107480261B (en) | Fine-grained face image fast retrieval method based on deep learning | |
| CN110909820B (en) | Image classification method and system based on self-supervision learning | |
| CN105184303B (en) | A kind of image labeling method based on multi-modal deep learning | |
| CN110826337A (en) | Short text semantic training model obtaining method and similarity matching algorithm | |
| CN103064903B (en) | Picture retrieval method and device | |
| CN108446334B (en) | Image retrieval method based on content for unsupervised countermeasure training | |
| CN112819023A (en) | Sample set acquisition method and device, computer equipment and storage medium | |
| CN107292349A (en) | The zero sample classification method based on encyclopaedic knowledge semantically enhancement, device | |
| CN113806746A (en) | Malicious code detection method based on improved CNN network | |
| CN113672693B (en) | Label recommendation method of online question-answering platform based on knowledge graph and label association | |
| CN110647919A (en) | Text clustering method and system based on K-means clustering and capsule network | |
| CN112434628A (en) | Small sample polarization SAR image classification method based on active learning and collaborative representation | |
| CN112925904B (en) | Lightweight text classification method based on Tucker decomposition | |
| CN112861524A (en) | Deep learning-based multilevel Chinese fine-grained emotion analysis method | |
| CN114048295A (en) | Cross-modal retrieval method and system for data processing | |
| CN113779283B (en) | Fine-grained cross-media retrieval method with deep supervision and feature fusion | |
| CN108920451A (en) | Text emotion analysis method based on dynamic threshold and multi-categorizer | |
| Lucchi et al. | Joint image and word sense discrimination for image retrieval | |
| CN111191033A (en) | Open set classification method based on classification utility | |
| US11914641B2 (en) | Text to color palette generator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |