CN113592031B - Image classification system, and method and device for identifying violation tool - Google Patents
Image classification system, and method and device for identifying violation tool Download PDFInfo
- Publication number
- CN113592031B CN113592031B CN202110945015.5A CN202110945015A CN113592031B CN 113592031 B CN113592031 B CN 113592031B CN 202110945015 A CN202110945015 A CN 202110945015A CN 113592031 B CN113592031 B CN 113592031B
- Authority
- CN
- China
- Prior art keywords
- module
- image
- layer
- features
- hierarchical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000000605 extraction Methods 0.000 claims abstract description 39
- 238000004364 calculation method Methods 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 15
- 230000000007 visual effect Effects 0.000 claims description 13
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 241000251128 Galeocerdo cuvier Species 0.000 description 13
- 241000283222 Physeter catodon Species 0.000 description 13
- 241000282326 Felis catus Species 0.000 description 12
- 241000282373 Panthera pardus Species 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 241000282376 Panthera tigris Species 0.000 description 6
- 238000013145 classification model Methods 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 241000282323 Felidae Species 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 241000251129 Galeocerdo Species 0.000 description 2
- 241000283249 Physeteridae Species 0.000 description 2
- 230000008485 antagonism Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 241000282324 Felis Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an image classification system, a method and a device for identifying a violation tool, wherein the image classification system comprises the following components: the system comprises a hierarchical feature extraction module, a hierarchical generation model and a hierarchical classification module, wherein the hierarchical generation model comprises at least one layer of generation module, and the hierarchical feature extraction module is used for extracting image features according to a target image; if the generating module is not the first layer generating module in the hierarchical generating model, determining the generating characteristics of the current layer according to the image characteristics, taking the upper layer artificial characteristics output by the upper layer generating module as a basic value, and taking the generating characteristics of the current layer as offset to form the artificial characteristics of the current layer; the hierarchical classification module is used for outputting an image classification result according to the current layer artificial characteristics. The upper layer artificial feature output by the generation module above the generation module serves as a basic value, so that the field drift of the current layer artificial feature calculated by the current generation module is reduced, and the finally obtained classification result is more accurate.
Description
Technical Field
The invention relates to the technical field of image classification, in particular to an image classification system, a method and a device for identifying violation tools.
Background
For training conventional image classification algorithms, such as deep convolutional neural networks, a large amount of tagged image data is required, which can lead to reduced recognition accuracy when new classes occur. In order to solve the problem, expert scientists at home and abroad propose a plurality of zero sample classification methods. In a zero sample classification scenario, the categories in the training set are known categories, the categories in the test set are unknown categories, and the two category sets do not overlap. The training set has only known classes of image data. At present, the zero sample classification method is mainly divided into three types, namely an embedding-based method, a model generation-based method and a knowledge graph-based method. The embedding-based method aims at learning a mapping function to map auxiliary information (such as attribute vectors and word vectors) and visual features into a public space, and the most recent label classification result is found after mapping the test image features during testing. The method is easy to cause a pivot point problem, a generation model is introduced in the zero sample classification field aiming at the problem, and auxiliary information is utilized to generate artificial features for unknown categories to solve the problem of data imbalance, however, the existing method based on the generation model utilizes a single network to generate the artificial features for the unknown categories in one step, and the method can cause the field drift problem, so that the final classification result is inaccurate.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the defect of inaccurate classification result calculated based on the generated model in the prior art, thereby providing an image classification system, a method and a device for identifying a violation tool.
The first aspect of the present invention provides an image classification system comprising: the system comprises a hierarchical feature extraction module, a hierarchical generation model and a hierarchical classification module, wherein the hierarchical generation model comprises at least one layer of generation module, and the hierarchical feature extraction module is used for extracting image features according to a target image and transmitting the image features to the generation module; if the generating module is not the first layer generating module in the hierarchical generating model, the generating module is used for determining the generating characteristics of the current layer according to the image characteristics, taking the upper layer artificial characteristics output by the upper layer generating module as a basic value, taking the generating characteristics of the current layer as offset, forming the artificial characteristics of the current layer, and transmitting the artificial characteristics of the current layer to the hierarchical classifying model; the hierarchical classification module is used for outputting an image classification result according to the current layer artificial characteristics.
Optionally, in the image classification system provided by the invention, the hierarchy classification module comprises at least one layer of classifier, the classifiers are in one-to-one correspondence with the generation modules, and the generation modules transmit the current layer of artificial features to the classifiers corresponding to the generation modules in the hierarchy classification module; if the classifier is not the first layer classifier in the hierarchical classification module, the classifier is used for obtaining a current layer classification result according to the current layer artificial feature, and obtaining an image classification result according to the current layer classification result and an upper layer classification result obtained by a previous layer classifier.
Optionally, in the image classification system provided by the invention, the classifier includes a plurality of node sets, each node set includes at least one node, and each node characterizes different image classification options; the node sets of the classifiers are in one-to-one correspondence with the nodes of the previous layer of the classifiers, and the nodes in the node sets are child nodes of the previous layer of the classifiers corresponding to the node sets.
Optionally, in the image classification system provided by the invention, the hierarchical feature extraction module comprises a basic feature extraction network and at least one layer of branch network, the branch networks are in one-to-one correspondence with the generation module, and the basic feature extraction network is used for extracting a basic feature set according to the target image and transmitting the basic feature set to the branch network; if the branch network is not the first layer branch network in the hierarchical feature extraction module, the branch network is used for acquiring an upper layer classification result obtained by an upper layer classifier, calculating the weight of each feature in the basic feature set according to the upper layer classification result, calculating the current layer feature according to the basic feature set and the weight of each feature, taking the current layer feature as an image feature, and transmitting the image feature to a generation module corresponding to the branch network.
Optionally, in the image classification system provided by the invention, the number of levels of the generation module in the hierarchy generation model is determined by the number of levels of the data set taxonomy structure corresponding to the target image.
Optionally, in the image classification system provided by the present invention, the branch network includes: the upper layer classification result conversion sub-module is used for expanding the upper layer classification result into visual characteristic dimension parameters; the abstract conversion sub-module is used for converting the features in the basic feature set into an abstract space to obtain abstract features; the weight calculation sub-module inputs the visual feature dimension parameters and the abstract features into the attention network to obtain weights of the features in the basic feature set; the visual feature calculation sub-module is used for performing dot product operation on the weights of the features and the abstract features to obtain local features; the global feature calculation module is used for calculating global features according to the basic features; and the current layer characteristic calculation module is used for superposing the local characteristic and the global characteristic to obtain the current layer characteristic.
Optionally, in the image classification system provided by the invention, the generating module generates the current layer artificial feature through the following formula: f's' gl ＝(1-α)×f gl +α×f′ g(l-1) Wherein f' g(l-1) Representing upper layer artificial features, f gl Representing the current layer generation characteristics, alpha being a super parameter.
The second aspect of the present invention provides a method for identifying a violation tool, comprising: acquiring an image to be classified; inputting the image to be classified into the image classification system provided by the first aspect of the invention to obtain the category of the tool in the image to be classified; if the category of the tool in the image to be classified belongs to the preset violation tool category, determining that the tool is a violation tool.
A third aspect of the present invention provides a violation tool identifying device, comprising: the image acquisition module is used for acquiring images to be classified; the classification module is used for inputting the images to be classified into the image classification system provided by the first aspect of the invention to obtain the types of tools in the images to be classified; and the violation tool identification module is used for judging the tool to be a violation tool if the class of the tool in the image to be classified belongs to the preset violation tool class.
A fourth aspect of the invention provides a computer device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to perform the method of identifying a violation tool as provided in the second aspect of the invention.
A fifth aspect of the present invention provides a computer readable storage medium having stored thereon computer instructions for causing a computer to perform the violation tool identifying method as provided in the second aspect of the present invention.
The technical scheme of the invention has the following advantages:
the image classification system, the violation tool identification method and the device provided by the invention comprise a hierarchical feature extraction module, a hierarchical generation model and a hierarchical classification module, wherein the hierarchical feature extraction module is used for firstly obtaining image features of a target image, then the hierarchical generation model is used for generating artificial features according to the image features, and finally a classification result is output according to the current layer artificial features through the hierarchical classification module, wherein the hierarchical generation model comprises a plurality of layers of generation modules, when the generation module for receiving the image features is not a first layer generation module in the hierarchical generation model, the generation module takes the upper layer artificial features output by the generation module above as basic values, the current layer generated features generated by the generation module above are taken as offset values, so that the current layer artificial features calculated by the generation module above are taken as basic values, and the field drift of the current layer artificial features calculated by the current generation module is reduced, so that the finally obtained classification result is more accurate.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a functional block diagram of one specific example of an image classification system in an embodiment of the invention;
FIG. 2 is a functional block diagram of one specific example of a generation module in an embodiment of the present invention;
FIG. 3 is a functional block diagram of one specific example of an image classification system in an embodiment of the invention;
FIG. 4 is a schematic diagram of a classification structure of a data set according to an embodiment of the present invention;
FIG. 5 is a schematic block diagram of a specific example of a hierarchical feature extraction module in an embodiment of the invention;
FIG. 6 is a flowchart of calculating image features of other branch networks except the first layer branch network in the hierarchical feature extraction module according to an embodiment of the present invention;
FIG. 7 is a flowchart of one specific example of a method of violation tool identification in an embodiment of the invention;
FIG. 8 is a schematic block diagram of one specific example of a violation tool identifying device in an embodiment of the present invention;
fig. 9 is a functional block diagram of a computer device provided in an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made apparent and fully in view of the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the description of the present invention, it should be noted that technical features of different embodiments of the present invention described below may be combined with each other as long as they do not collide with each other.
An embodiment of the present invention provides an image classification system, as shown in fig. 1, including: the system comprises a hierarchical feature extraction module, a hierarchical generation model and a hierarchical classification module, wherein the hierarchical generation model comprises at least one layer of generation module.
The hierarchical feature extraction module is used for extracting image features according to the target image and transmitting the image features to the generation module.
In an alternative embodiment, the target image includes an object to be classified, and the image classification system is configured to classify the object to be classified in the target image, so as to identify the object to be classified in the target image. In an alternative embodiment, the object to be classified may be an animal, plant, tool, etc.
In an alternative embodiment, the hierarchical feature extraction module uses the image features extracted from the target image as features of the objects to be classified in the target image, and the extracted features are different for different objects to be classified.
In an alternative embodiment, after the hierarchical feature extraction module obtains the image features, the image features are transmitted to a generation module in the hierarchical generation model.
If the received image feature generation module is not the first layer generation module in the hierarchical generation model, the generation module is used for determining the current layer generation feature according to the image feature, taking the upper layer artificial feature output by the upper layer generation module as a basic value, taking the current layer generation feature as an offset, forming the current layer artificial feature, and transmitting the current layer artificial feature to the hierarchical classification model.
The hierarchical classification module is used for outputting an image classification result according to the current layer artificial characteristics. In an alternative embodiment, the hierarchical classification module includes a classifier, and the image classification result can be obtained through the classifier.
The image classification system provided by the embodiment of the invention comprises a hierarchical feature extraction module, a hierarchical generation module and a hierarchical classification module, wherein the hierarchical feature extraction module is used for firstly obtaining the image features of a target image, then the hierarchical generation module is used for generating artificial features according to the image features, and finally the hierarchical classification module is used for outputting classification results according to the current layer artificial features, wherein the hierarchical generation module comprises a plurality of layers of generation modules, when the generation module for receiving the image features is not the first layer generation module in the hierarchical generation module, the generation module takes the upper layer artificial features output by the generation module of the previous layer as basic values, takes the current layer generated features generated by the generation module of the previous layer as offset values, so that the current layer artificial features are calculated by the generation module of the previous layer as basic values, and the field drift of the current layer artificial features calculated by the current generation module is reduced, so that the finally obtained classification results are more accurate.
In an alternative embodiment, in the hierarchical generation model, except for the first layer generation module, other generation modules include an antagonism generation network and a feature synthesis submodule, where the antagonism generation network is used to generate the current layer generation feature, and the feature synthesis submodule is used to use the upper layer artificial feature output by the upper layer generation module as a basic value, use the current layer generation feature as an offset, form the current layer artificial feature, and input the current layer artificial feature into the hierarchical classification model.
In an alternative embodiment, in the hierarchical generation model, a countermeasure generation network is included in the first layer generation module for generating the current layer characteristics.
In an alternative embodiment, if the generation module that receives the image feature is the first generation module in the hierarchical generation model, the current generation feature determined by the generation module according to the image feature is determined to be the current artificial feature.
In an alternative embodiment, as shown in FIG. 2, the countermeasure generation network includes a generator for generating a vector (a) according to category attributes y ) And calculating a synthetic feature from noise (N (0, 1)) randomly generated from the normal distribution; the discriminator is used for judging whether the input features are real features or synthesized features, and when training the countermeasure generation network, the discriminator is used for guiding the training of the generator. The current layer generation features in the above embodiments are calculated by the generator.
In an alternative embodiment, the current layer artificial feature is calculated by the following formula:
f′ gl ＝(1-α)×f gl +α×f′ g(l-1) ，
wherein f' g(l-1) Representing upper layer artificial features, f gl Representing the current layer generation characteristics, alpha being a super parameter.
In an alternative embodiment, as shown in FIG. 3, the hierarchical classification module includes at least one layer of classifiers (classifiers l ) The classifier corresponds to the generation module one by one.
In an alternative embodiment, the number of levels of the generation module in the hierarchical generation model is determined by the number of levels of the data set taxonomy structure corresponding to the target image.
For example, if the target image includes an animal, the data set taxonomy structure corresponding to the target image is a data set taxonomy structure constructed according to animal taxonomies. As shown in fig. 4, if the data set taxonomic structure includes three levels, the first level includes tiger shark family, sperm whale family, feline family, the second level includes tiger shark genus, sperm whale genus, cat genus, leopard genus, the third level pair includes tiger shark, sperm whale, cat, tiger, leopard, the number of levels of the classifier and the generation module is 3.
In an alternative embodiment, the classifier includes a plurality of node sets, each node set including at least one node, each node characterizing a different image classification option. The node sets of the classifiers are in one-to-one correspondence with the nodes of the previous layer of the classifiers, and the nodes in the node sets are child nodes of the previous layer of the classifiers corresponding to the node sets. The number of nodes corresponding to each classifier and the relation between each node and the upper node are determined through the data set taxonomy structure corresponding to the target image.
If the data set taxonomic structure corresponding to the target image is shown in fig. 4, in the image classification system, the first layer classifier includes three nodes corresponding to the tiger shark family, the sperm whale family and the cat family respectively, the second layer classifier includes four nodes corresponding to the tiger shark family, the sperm whale family, the cat genus and the leopard genus respectively, and the third layer classifier includes five nodes corresponding to the tiger shark family, the sperm whale family, the cat family, the tiger and the leopard genus respectively. Moreover, the tiger shark genus in the second layer classifier is a child node of the tiger shark family in the first layer classifier, the sperm whale genus in the second layer classifier is a child node of the sperm whale family in the first layer classifier, and the cat genus and leopard genus in the second layer classifier are child nodes of the cat family in the first layer classifier; the tiger sharks in the third-layer classifier are child nodes of the tiger sharks in the second-layer classifier, the sperm whales in the third-layer classifier are child nodes of the sperm whales in the second-layer classifier, the cats and tigers in the third-layer classifier are child nodes of the cat genus in the second-layer classifier, and the leopards in the third-layer classifier are child nodes of the leopard genus in the second-layer classifier.
In the image classification system provided by the embodiment of the invention, the hierarchical structure of the hierarchical generation model and the hierarchical classification module is constructed according to the data set classification structure, so that the image classification system can mine and utilize the shared information of the known category and the unknown category in the knowledge graph when classifying the target image.
In an optional embodiment, in the image classification system provided by the embodiment of the present invention, after the generating module generates the current layer artificial feature, a manner of determining the image classification result is as follows:
the generation module transmits the current layer artificial feature to the classifier corresponding to the generation module in the hierarchical classification module.
In an alternative embodiment, the artificial features generated by each layer of generating module are different, and each layer of classifier has different classification options when classifying the target object, so the generating module needs to input the current layer of artificial features into the classifier corresponding to the current layer of artificial features.
If the classifier is not the first layer classifier in the hierarchical classification module, the classifier is used for obtaining a current layer classification result according to the current layer artificial feature, and obtaining an image classification result according to the current layer classification result and an upper layer classification result obtained by a previous layer classifier.
In an alternative embodiment, each classifier corresponds to different classification options, the output image classification result is a probability value corresponding to each classification option, if the current classifier is not the first layer classifier, the current classifier calculates the probability value of each classification option corresponding to the current classifier by combining the self classification result and the probability value of each classification option in the upper layer classifier when calculating the probability value corresponding to each option, and the probability value of each classification option corresponding to the current classifier is the image classification result obtained by the current classifier.
The classification options corresponding to the previous layer of classifier include probabilities corresponding to tiger shark, sperm whale, cat and leopard respectively, the self classification result obtained by the next layer of classifier is probabilities corresponding to tiger shark, sperm whale, cat, tiger and leopard respectively, then the probabilities corresponding to tiger shark and the probabilities corresponding to tiger shark are combined to obtain the final probability of tiger shark, the probabilities corresponding to sperm whale and the probabilities corresponding to sperm whale are combined to obtain the final probability of sperm whale, the probabilities corresponding to cat and the probabilities corresponding to cat are combined to obtain the final probability of cat, the probabilities corresponding to tiger and the probabilities corresponding to cat are combined to obtain the final probability of tiger, and the probabilities corresponding to leopard are combined to obtain the final probability of leopard.
In an alternative embodiment, when the probability value of each classification option in the self classification result is combined with the probability value of the parent node of each classification option, the sum may be obtained, the average may be obtained, and other manners may be adopted.
In an alternative embodiment, if the classifier is the first layer classifier in the hierarchical classification module, the probability value of each classification option in the self classification result is determined as the image classification result.
In the technical scheme provided by the embodiment of the invention, the classifier comprises a plurality of node sets, the node sets comprise a plurality of nodes, different nodes respectively represent different image classification options, each node of the lower-layer classifier is a child node of the upper-layer classifier adjacent to the node, namely, in the embodiment of the invention, the dimension of the classification option is expanded according to the data set taxonomy structure, and the larger the probability value of the classification option corresponding to a certain node in the previous-layer classifier is, the more likely the classification option with the highest probability value obtained by the current-layer classifier is the child node thereof. In the embodiment of the invention, the classification of the target image is performed step by step, so that the classification process is more refined, and the obtained classification result is more accurate.
In an alternative embodiment, as shown in fig. 3, the hierarchical feature extraction module includes a basic feature extraction network and at least one layer of branch network, where the branch networks are in one-to-one correspondence with the generation module. The method for determining the number of levels of the branch network is the same as the method for determining the number of levels of the generation module, and details of the method are described in the above embodiments, and are not repeated here.
In an alternative embodiment, the underlying feature extraction network may employ the first 41 layers of ResNet-50.
The basic feature extraction network is used for extracting a basic feature set according to the target image and transmitting the basic feature set to the branch network. In an alternative embodiment, the basic feature set entered into all the branched networks is the same.
If the branch network is not the first layer branch network in the hierarchical feature extraction module, the branch network is used for acquiring an upper layer classification result obtained by an upper layer classifier, calculating the weight of each feature in the basic feature set according to the upper layer classification result, calculating the current layer feature according to the basic feature set and the weight of each feature, taking the current layer feature as an image feature, and transmitting the image feature to a generation module corresponding to the branch network.
As described in the above embodiment, different classifiers are used to implement classification at different levels, and the focus of the used features is different for classification at different levels, so in the embodiment of the invention, different branching networks are used to generate different image features respectively, and then the different image features are input into corresponding generation modules.
In an optional embodiment, because the upper layer classification option and the current layer classification option have an upper-lower relationship, the current layer classification option is an option refined by the upper layer classification option, when the weights of the features in the basic feature set are calculated according to the upper layer classification result, the relevance of the features and the upper layer classification options can be determined first, the features corresponding to the classification options with larger probability values in the upper layer classification result are given a larger weight, and the features corresponding to the classification options with smaller probability values in the upper layer classification result are given a smaller weight.
The image characteristics obtained by the image classification system provided by the embodiment of the invention pay more attention to local characteristics, and the classification results obtained by the classifiers of each layer have smaller deviation.
In an alternative embodiment, the branch networks of the other layers except for the first layer branch network in the hierarchical feature extraction module respectively include:
an upper layer classification result conversion sub-module for expanding the upper layer classification result into visual feature dimension parametersIn an alternative embodiment, the generated classification result may be input into a pre-trained neural network model, so as to obtain the dimension parameters of the visual features.
An abstract conversion sub-module for converting the features in the basic feature set into an abstract space to obtain abstract featuresWherein, in an alternative embodiment, the characteristics in the basic characteristic set can be input into a pre-trained neural network model, so as to obtain abstract characteristics.
The weight calculation sub-module inputs the visual feature dimension parameters and the abstract features into the attention network to obtain weights of the features in the basic feature set:
the visual feature calculation sub-module is used for performing dot product operation on the weights of the features and the abstract features to obtain local features f l 。
A global feature calculation module for calculating global features according to the basic featuresWherein, in an alternative embodiment, the basic characteristics can be input into a pre-trained neural network model, so as to obtain global characteristics.
The current layer feature calculation module is used for superposing the local features and the global features to obtain current layer features: f (f) rl . In an alternative embodiment, the part will beThe feature and global feature superposition may be a calculation of the mean of the local and global features.
In an alternative embodiment, the flow of computing the current layer features by the branch networks of the layers other than the first layer branch network in the hierarchical feature extraction module is shown in fig. 5.
In an alternative embodiment, the first layer branch network in the hierarchical feature extraction module includes:
an abstract conversion sub-module for converting the features in the basic feature set into an abstract space to obtain abstract featuresWherein, represent.
A global feature calculation module for calculating global features according to the basic featuresWherein, represent.
The current layer feature calculation module is used for superposing the abstract features and the global features to obtain current layer features: f (f) rl . In an alternate embodiment, superimposing the abstract and global features may be computing a mean of the abstract and global features.
In an alternative embodiment, the hierarchical feature extraction module may be trained separately, as shown in fig. 6, when the hierarchical feature extraction module is trained, the global features obtained by the branch networks are classified to obtain a classification result, if the current branch network is not the last branch network, the classification result is input into the next branch network, the step of classifying the global features obtained by the branch networks to obtain a classification result is repeatedly performed in the next branch network until the current branch network is the last branch network, the current classification result is determined to be the final classification result, if the accuracy of the final classification result is less than a preset value, the parameters in each branch network are corrected according to the final classification result, and if the accuracy of the final classification result is greater than or equal to the preset value, the hierarchical extraction model is used as a part of the image classification system to obtain the image features.
The embodiment of the invention also provides a method for identifying the violation tool, which comprises the following steps as shown in fig. 7:
step S21: and acquiring an image to be classified. In an alternative embodiment, if the illegal tools on the power grid operation site need to be identified, the image acquisition equipment arranged on the power grid operation site can acquire a power grid operation site picture, wherein the power grid operation site picture comprises the tools.
Step S22: the image to be classified is input into the image classification system to obtain the category of the tool in the image to be classified, and the image classification system for identifying the offending tool is provided in any one of the embodiments.
If the category of the tool in the image to be classified belongs to the preset violation tool category, judging that the tool in the image to be classified is a violation tool; if the category of the tool in the image to be classified does not belong to the preset illegal tool category, judging that the tool in the image to be classified is not the illegal tool.
In the embodiment of the invention, various illegal tool types can be preset, and because tools required to be used in different application scenes are different, tools forbidden to be used are also different, and therefore, the tool types contained in the preset illegal tool types are different for different application scenes.
In an alternative embodiment, when the image classification system is used to identify the class of the offending tool, the number of levels of the branch network in the hierarchical feature extraction module, the number of levels of the generation module in the hierarchical generation model, the number of levels of the classifier in the hierarchical classification module, and the classification options of each classifier in the hierarchical classification model in the image classification system are determined according to the data set taxonomy structure of the tool.
In an alternative embodiment, the image classification system provided in the above embodiment classifies the violation tools in the images to be classified according to the following procedures:
firstly, extracting image features of an image to be classified by a hierarchical feature extraction module, wherein the image features are features of tools in the image to be classified.
And inputting the image features into the hierarchical generation model, and if the generation module of the received image features is not the first generation module in the hierarchical generation model, taking the upper artificial features output by the upper generation module as basic values, taking the current generation features as offset values, forming the current artificial features, and transmitting the current artificial features to the hierarchical classification model.
And finally, the hierarchical classification module outputs the violation tool class according to the manual characteristics of the current layer.
In an alternative embodiment, the process of classifying the violation tool in the image to be classified by the image classification system provided in the foregoing embodiment may further be:
step one, extracting a basic feature set according to a target image through a basic feature extraction network in the hierarchical feature extraction module, and respectively inputting the basic feature set into each branch network. The first layer branch network calculates the basic feature set by adopting different functions to obtain abstract features and global features, the abstract features and the global features are overlapped to obtain current layer features corresponding to the first layer branch network, and the current layer features are input into the first layer generation module.
Step two, the first layer generation module obtains the current layer artificial feature according to the current layer feature, and inputs the current layer artificial feature into the first layer classifier.
Thirdly, the first layer classifier obtains probability values of classification options corresponding to the first layer classifier according to the current layer artificial feature.
Step four, the next layer branch network receives probability values of classification options obtained by the previous layer classifier, and expands the probability values of the classification options obtained by the previous layer classifier into visual feature dimension parameters through a linear conversion function; converting the features in the basic feature set into an abstract space to obtain abstract features; inputting the visual feature dimension parameters and the abstract features into an attention network to obtain weights of the features in the basic feature set; performing dot product operation on the weights of the features and the abstract features to obtain local features; calculating global features according to the basic features; superposing the local features and the global features to obtain current layer features; and determining the current layer characteristics as graphic characteristics and inputting the graphic characteristics into a corresponding generation module.
And fifthly, determining the current layer generation feature by the generation module of the received image feature according to the image feature, taking the upper layer artificial feature output by the generation module of the upper layer as a basic value, taking the current layer generation feature as an offset, forming the current layer artificial feature, and transmitting the current layer artificial feature to the hierarchical classification model.
Step six, the classifier which receives the current layer artificial feature obtains a current layer classification result according to the current layer artificial feature, and obtains probability values of classification options according to the current layer classification result and an upper layer classification result obtained by a last layer classifier.
Judging whether a next-layer branch network exists or not, generating a module or a classifier, if the next-layer branch network exists or generating the module or the classifier, repeating the steps four, five and six until the next-layer branch network does not exist or generating the module or the classifier, outputting the probability value of each classification option corresponding to the current classifier, and determining the type of the violation tool corresponding to the classification option with the largest probability value as the type of the violation tool in the image to be classified.
The embodiment of the invention also provides a device for identifying the violation tool, as shown in fig. 8, comprising:
the image obtaining module 31 is configured to obtain the image to be classified, and the details of the image obtaining module are described in the above embodiment in step S21, which is not described herein.
The classifying module 32 is configured to input the image to be classified into the image classifying system to obtain the category of the tool in the image to be classified, and the detailed description of step S22 in the above embodiment is omitted here.
The offending tool identification module 33, if the category of the tool in the image to be classified belongs to the preset offending tool category, the offending tool identification module 33 is configured to determine that the tool is an offending tool, and details of the foregoing description in the foregoing method embodiment are not repeated herein.
The embodiment of the present invention provides a computer device, as shown in fig. 9, which mainly includes one or more processors 41 and a memory 42, and in fig. 9, one processor 41 is taken as an example.
The computer device may further include: an input device 43 and an output device 44.
The processor 41, the memory 42, the input device 43 and the output device 44 may be connected by a bus or otherwise, for example in fig. 9.
The processor 41 may be a central processing unit (Central Processing Unit, CPU). The processor 41 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Memory 42 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created from the use of the violation tool identifying device, etc. In addition, memory 42 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 42 may optionally include memory located remotely from processor 41, which may be connected to the violation tool identifying device via a network. The input device 43 may receive a user entered calculation request (or other numeric or character information) and generate key signal inputs related to the violation tool identifying device. The output device 44 may include a display device such as a display screen for outputting the calculation result.
Embodiments of the present invention provide a computer readable storage medium storing computer instructions, the computer storage medium storing computer executable instructions that are capable of performing the method for identifying a violation tool in any of the method embodiments described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), or a Solid State Drive (SSD); the storage medium may also comprise a combination of memories of the kind described above.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.
Claims (9)
1. An image classification system, comprising: the system comprises a hierarchical feature extraction module, a hierarchical generation model and a hierarchical classification module, wherein the hierarchical generation model comprises at least one layer of generation module,
the hierarchical feature extraction module is used for extracting image features according to the target image and transmitting the image features to the generation module;
if the generating module is not the first layer generating module in the hierarchical generating model, the generating module is used for determining the current layer generating characteristic according to the image characteristic, taking the upper layer artificial characteristic output by the upper layer generating module as a basic value, taking the current layer generating characteristic as an offset, forming the current layer artificial characteristic, and transmitting the current layer artificial characteristic to the hierarchical classifying model;
the hierarchical classification module is used for outputting an image classification result according to the current layer artificial feature;
the hierarchical classification module comprises at least one layer of classifier which is in one-to-one correspondence with the generation module,
the generation module transmits the current layer artificial feature to a classifier corresponding to the generation module in the hierarchical classification module;
if the classifier is not the first layer classifier in the hierarchical classification module, the classifier is used for obtaining a current layer classification result according to the current layer artificial feature, and obtaining the image classification result according to the current layer classification result and an upper layer classification result obtained by a previous layer classifier;
the hierarchical feature extraction module comprises a basic feature extraction network and at least one layer of branch network, the branch network corresponds to the generation module one by one,
the basic feature extraction network is used for extracting a basic feature set according to the target image and transmitting the basic feature set to the branch network;
if the branch network is not the first layer branch network in the hierarchical feature extraction module, the branch network is used for acquiring an upper layer classification result obtained by an upper layer classifier, calculating weights of features in the basic feature set according to the upper layer classification result, calculating a current layer feature according to the basic feature set and the weights of the features, taking the current layer feature as the image feature, and transmitting the image feature to a generation module corresponding to the branch network.
2. The image classification system of claim 1, wherein,
the classifier comprises a plurality of node sets, each node set comprises at least one node, and each node respectively represents different image classification options;
the node sets of the classifiers are in one-to-one correspondence with the nodes of the previous layer of the classifiers, and the nodes in the node sets are child nodes of the previous layer of the classifiers corresponding to the node sets.
3. An image classification system according to claim 1 or 2, characterized in that,
the number of levels of the generation modules in the hierarchy generation model is determined by the number of levels of the data set taxonomy structure corresponding to the target image.
4. The image classification system of claim 1, wherein the branching network comprises:
the upper layer classification result conversion sub-module is used for expanding the upper layer classification result into visual characteristic dimension parameters;
the abstract conversion sub-module is used for converting the features in the basic feature set into an abstract space to obtain abstract features;
the weight calculation sub-module inputs the visual feature dimension parameters and the abstract features into an attention network to obtain weights of all features in the basic feature set;
the visual feature calculation sub-module is used for performing dot product operation on the weights of the features and the abstract features to obtain local features;
the global feature calculation module is used for calculating global features according to the basic features;
and the current layer feature calculation module is used for superposing the local features and the global features to obtain the current layer features.
5. The image classification system of claim 1, wherein the generation module generates the current layer artificial feature by:
f′ g ＝(1-α)×f gl +α×f′ g(l-1) ，
wherein f' g(l-1) Representing upper layer artificial features, f gl Representing the current layer generation characteristics, alpha being a super parameter.
6. A method of identifying a violation tool, comprising:
acquiring an image to be classified;
inputting the image to be classified into the image classification system according to any one of claims 1-5 to obtain the category of the tool in the image to be classified;
and if the category of the tool in the image to be classified belongs to the preset violation tool category, judging that the tool is a violation tool.
7. A violation tool identifying device, comprising:
the image acquisition module is used for acquiring images to be classified;
the classification module is used for inputting the image to be classified into the image classification system according to any one of claims 1-5 to obtain the category of the tool in the image to be classified;
and the violation tool identification module is used for judging that the tool is a violation tool if the class of the tool in the image to be classified belongs to the preset violation tool class.
8. A computer device, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to perform the violation tool identifying method of claim 6.
9. A computer-readable storage medium storing computer instructions for causing the computer to perform the violation tool recognition method of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110945015.5A CN113592031B (en) | 2021-08-17 | 2021-08-17 | Image classification system, and method and device for identifying violation tool |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110945015.5A CN113592031B (en) | 2021-08-17 | 2021-08-17 | Image classification system, and method and device for identifying violation tool |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113592031A CN113592031A (en) | 2021-11-02 |
| CN113592031B true CN113592031B (en) | 2023-11-28 |
Family
ID=78258389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110945015.5A Active CN113592031B (en) | 2021-08-17 | 2021-08-17 | Image classification system, and method and device for identifying violation tool |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113592031B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108765512A (en) * | 2018-05-30 | 2018-11-06 | 清华大学深圳研究生院 | A kind of confrontation image generating method based on multi-layer feature |
| CN110309861A (en) * | 2019-06-10 | 2019-10-08 | 浙江大学 | A kind of multi-modal mankind's activity recognition methods based on generation confrontation network |
| CN110543563A (en) * | 2019-08-20 | 2019-12-06 | 暨南大学 | Hierarchical text classification method and system |
| CN111737521A (en) * | 2020-08-04 | 2020-10-02 | 北京微播易科技股份有限公司 | Video classification method and device |
| CN112183672A (en) * | 2020-11-05 | 2021-01-05 | 北京金山云网络技术有限公司 | Image classification method, and training method and device of feature extraction network |
-
2021
- 2021-08-17 CN CN202110945015.5A patent/CN113592031B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108765512A (en) * | 2018-05-30 | 2018-11-06 | 清华大学深圳研究生院 | A kind of confrontation image generating method based on multi-layer feature |
| CN110309861A (en) * | 2019-06-10 | 2019-10-08 | 浙江大学 | A kind of multi-modal mankind's activity recognition methods based on generation confrontation network |
| CN110543563A (en) * | 2019-08-20 | 2019-12-06 | 暨南大学 | Hierarchical text classification method and system |
| CN111737521A (en) * | 2020-08-04 | 2020-10-02 | 北京微播易科技股份有限公司 | Video classification method and device |
| CN112183672A (en) * | 2020-11-05 | 2021-01-05 | 北京金山云网络技术有限公司 | Image classification method, and training method and device of feature extraction network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113592031A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lucchi et al. | Are spatial and global constraints really necessary for segmentation? | |
| CN110163258B (en) | Zero sample learning method and system based on semantic attribute attention redistribution mechanism | |
| US11687716B2 (en) | Machine-learning techniques for augmenting electronic documents with data-verification indicators | |
| CN110728295B (en) | Semi-supervised landform classification model training and landform graph construction method | |
| CN110059646B (en) | Method for training action planning model and target searching method | |
| JP2020038661A (en) | Learning method and learning device for detecting lane by using lane model, and test method and test device using the same | |
| CN112464717A (en) | Remote sensing image target detection method, system, electronic equipment and storage medium | |
| CN112989942A (en) | Target instance segmentation method based on traffic monitoring video | |
| CN114048468A (en) | Intrusion detection method, intrusion detection model training method, device and medium | |
| CN113298186A (en) | Network abnormal flow detection method for confluent flow model confrontation generation network and clustering algorithm | |
| CN116934385B (en) | Construction method of user loss prediction model, user loss prediction method and device | |
| Kumar et al. | Feature relevance analysis for 3D point cloud classification using deep learning | |
| CN107451617B (en) | Graph transduction semi-supervised classification method | |
| CN111144462A (en) | Unknown individual identification method and device for radar signals | |
| CN117197591B (en) | Data classification method based on machine learning | |
| CN113592031B (en) | Image classification system, and method and device for identifying violation tool | |
| Liu et al. | Learning multiple gaussian prototypes for open-set recognition | |
| CN113094533A (en) | Mixed granularity matching-based image-text cross-modal retrieval method | |
| US20220139069A1 (en) | Information processing system, information processing method, and recording medium | |
| CN107169450A (en) | The scene classification method and system of a kind of high-resolution remote sensing image | |
| Hu et al. | Panoptic edge detection | |
| CN114154572A (en) | Heterogeneous data centralized access analysis method based on heterogeneous platform | |
| Li et al. | Semi-supervised implicit scene completion from sparse LiDAR | |
| Yu et al. | Weakly supervised learning of point-level annotation for coral image segmentation | |
| CN104778468A (en) | Image processing device, image processing method and monitoring equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |