CN113191149A - Method for automatically extracting information of Internet of things equipment - Google Patents
Method for automatically extracting information of Internet of things equipment Download PDFInfo
- Publication number
- CN113191149A CN113191149A CN202110516557.0A CN202110516557A CN113191149A CN 113191149 A CN113191149 A CN 113191149A CN 202110516557 A CN202110516557 A CN 202110516557A CN 113191149 A CN113191149 A CN 113191149A
- Authority
- CN
- China
- Prior art keywords
- information
- text
- word
- vector
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000003062 neural network model Methods 0.000 claims abstract description 7
- 239000000284 extract Substances 0.000 claims abstract description 4
- 239000013598 vector Substances 0.000 claims description 67
- 238000012549 training Methods 0.000 claims description 9
- 238000012790 confirmation Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000002457 bidirectional effect Effects 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000000306 recurrent effect Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000006403 short-term memory Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 2
- 230000007787 long-term memory Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000015654 memory Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Machine Translation (AREA)
Abstract
本发明公开了一种自动化提取物联网设备信息的方法，包括：步骤一：使用深度神经网络模型，训练得到设备类型分类器。给定一个应用层报文，通过训练好的分类器，得到设备类型信息。步骤二：在步骤一的应用层报文的基础上，利用命名实体识别技术，提取报文中物联网设备厂商的文字，作为设备厂商信息。步骤三：基于步骤二获得的设备厂商信息，在其周围文字，利用相似度计算，提取超过阈值的文字，作为产品型号信息。步骤四：针对物联网设备的不同种类信息的不同特点，本方法自动提取应用层报文中的设备类型、设备厂商以及产品型号。本方法部署方便，无需人工编写规则，是一种低成本、高效的物联网设备信息提取技术。
The invention discloses a method for automatically extracting device information of the Internet of Things, comprising: step 1: using a deep neural network model to train to obtain a device type classifier. Given an application layer packet, the device type information is obtained through the trained classifier. Step 2: On the basis of the application layer message in step 1, using the named entity recognition technology, the text of the IoT device manufacturer in the message is extracted as the device manufacturer information. Step 3: Based on the equipment manufacturer information obtained in Step 2, the texts around it are calculated by similarity, and the texts exceeding the threshold are extracted as product model information. Step 4: According to the different characteristics of different types of information of IoT devices, the method automatically extracts the device type, device manufacturer and product model in the application layer message. The method is convenient to deploy, does not need to manually write rules, and is a low-cost and efficient technology for extracting information from Internet of Things devices.
Description
Technical Field
The invention relates to the field of information security, in particular to a method for automatically extracting information of equipment of the Internet of things.
Background
Hundreds of millions of internet of things devices are accessed in a network space, and the variety of the internet of things devices is various, including office equipment, monitoring equipment, network equipment, industrial control equipment and the like. The internet of things equipment is the most important asset in the network space, and the detection, discovery and identification of the internet of things equipment in the network space become effective means for guaranteeing the safety of key infrastructure of the network space. The information of the internet of things records the type of a certain device, the manufacturer from the certain device, the specific product type number and other related information, and the information of the internet of things is important for security audit and security defense. At present, the existing method for extracting the information of the internet of things equipment depends on manual writing rules, or the range of extracting the information is limited, so that the method has certain limitations in the aspects of large-scale application and field deployment.
Therefore, when various internet of things devices exist in a network space, including a router, a network camera, a network printer and the like, how to effectively and automatically extract triples (device types, device manufacturers and product models) in application layer message information has application value.
Disclosure of Invention
The invention aims to provide a method for automatically extracting information of equipment of the Internet of things, so as to solve the problems in the technology in the background discussion.
The technical scheme of the invention is as follows:
a method for automatically extracting Internet of things equipment information comprises the following steps:
the method comprises the following steps: the determination of the device type information includes: step a, preprocessing message information of an application layer, deleting interference content, and converting slogans into text formats as input of all subsequent steps after a preprocessing module is completed; b, converting characters in a plain text format into word vectors, and training to obtain an equipment type classifier; step c, processing the application layer message to obtain equipment type information;
step two: the confirmation of the equipment manufacturer information comprises the following steps: step d, utilizing a named entity identification technology to identify the entity to which the text belongs; step e, obtaining equipment manufacturer information by using a recurrent neural network model;
step three: the confirmation of the product model information includes: extracting characters exceeding a threshold value by utilizing similarity calculation near the character information of the equipment manufacturer to obtain product model information;
step four, the confirmation of the information of the equipment of the Internet of things comprises the following steps: and combining the three steps to obtain the information of the equipment of the Internet of things, namely (equipment type, equipment manufacturer and product model).
Preferably, the pretreatment in step a comprises the steps of: a1, deleting the error state code of the application layer; a2, deleting irrelevant contents of the hypertext markup language; a3, removing special characters; a4, deleting time stamps, numbers, punctuation and stop words; a5, extracting a plain text from the rest message content, splitting the plain text into single characters, and performing word marking;
the step b specifically comprises the following steps: processing training data by using Word2Vec to obtain a pre-trained model, converting characters in a plain text format into Word vectors, and training to obtain a classifier of the equipment type by using the bidirectional long-short term memory network model based on an attention mechanism and taking the Word vectors as input;
the step c specifically comprises the following steps: giving an application layer message information, converting the application layer message information into a text mark and a vector mark as the input of a model; and the classifier gives the judgment of the type of the equipment of the Internet of things and provides a label of the equipment type: (device type, #, #).
Preferably, step d specifically includes: the application layer message information processed in the first step becomes a plain text, the category to which each word belongs is identified and marked as V and O, wherein V represents the category of equipment manufacturers, and O represents other categories;
the step e specifically comprises the following steps: carrying out three different vectorization on the plain text information in the step one, wherein the vectorization comprises a word vector, a letter vector and a mixed vector; using a gate control circulation unit model to express the letter vector of a word, and finally combining the word vector and the letter vector to be used as a single sequence vector, namely mixed vector expression; taking the mixed vector representation as the input of each gated cyclic unit, and training a cyclic neural network model so as to mark each character in the plain text information in the step one; searching a text marked as V, serving as a manufacturer of the Internet of things equipment, and providing a label of the equipment manufacturer: (#, equipment manufacturer, #).
Preferably, the third step is specifically: setting a window with the length of W based on the equipment manufacturer category V in the step two, finding all the characters appearing in the window, and generating a candidate set B; performing letter-level word vector representation and general word vector representation on each character in the set B; the known product model name of the internet of things is used as a set A, vector representation of characters in a set B is compared with vector representation of characters in the set A, if the similarity exceeds a threshold value T, the characters are used as the product model of the equipment, and a label of the product model is obtained: (#, #, product type).
The invention has the beneficial effects that: the method provides an effective automation technology, and the information of the Internet of things equipment (equipment type, equipment manufacturer and product model) is automatically and effectively extracted from the application layer message. The method is convenient to deploy, does not need to compile rules manually, and is a low-cost and high-efficiency Internet of things equipment information extraction technology.
Drawings
Fig. 1 is a flowchart of a method for automatically extracting information of an internet of things device according to an embodiment of the present invention;
FIG. 2 is a flowchart of extracting device types using a classifier according to an embodiment of the present invention;
fig. 3 is a model structure diagram of an internet of things device type according to an embodiment of the present invention;
fig. 4 is a diagram illustrating extraction of information of a device manufacturer of the internet of things by using a named entity recognition technology according to an embodiment of the present invention.
Fig. 5 is a flowchart of extracting a product model based on a device manufacturer and an existing product information set according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
Fig. 1 is a flowchart of a method for automatically extracting information of an internet of things device. Specifically, a method for automatically extracting information of an internet of things device includes:
the method comprises the following steps: and training to obtain the device type classifier by using the deep neural network model. And giving an application layer message, and obtaining the equipment type information through a trained classifier.
Step two: and on the basis of the application layer message in the step one, extracting the characters of the equipment manufacturer of the Internet of things in the message as equipment manufacturer information by using a named entity identification technology.
Step three: and based on the equipment manufacturer information obtained in the step two, extracting characters exceeding a threshold value from the characters around the equipment manufacturer information by utilizing similarity calculation to serve as product model information.
Step four: aiming at different characteristics of different kinds of information of the Internet of things equipment, the method automatically extracts the equipment type, equipment manufacturer and product model in the application layer message.
FIG. 2 is a flow chart of extracting device types using a classifier.
The first step specifically comprises the following steps:
aiming at message information of an application layer, the method needs to be preprocessed and the interference content is deleted, and the method comprises the following steps: (1) the error status codes of the application layer, e.g. 4XX, 5XX, are deleted. 400 indicates an error request and 500 indicates an internal server error; (2) irrelevant content such as tags, CSS, and JS in the hypertext markup language (HTML) is deleted. Specifically, these tags are surrounded by sharp brackets, such as < br >; (3) removing special characters, such as "$", "%"; (4) deleting timestamps, numbers, punctuation and stop words; (5) in the rest of the message contents, the plain text is extracted and split into single characters, which is called word tagging. And after the preprocessing module is finished, converting the slogan into a text format as the input of all the subsequent steps.
For a word in plain text format, this step will convert it into a word vector. Specifically, the method uses Word2Vec to process training data to obtain a pre-trained model, and words in a plain text format are converted into Word vectors. In the step, a Bidirectional Long Short-Term Memory network model (all called as extension-Based Bidirectional Short-Term Memory Networks) Based on an Attention mechanism is utilized, and a word vector is used as input to train to obtain a classifier of the equipment type. Giving an application layer message information, converting the application layer message information into a text mark and a vector mark as the input of a model; and the classifier gives a decision on the type of the internet of things device, i.e. it provides a label in the form of (device type, #, #) for it.
Fig. 3 is a model structure diagram of the types of devices in the internet of things in the embodiment of the present invention. The attention mechanism model contains 5 parts: (1) an input layer: inputting a statement into the model through the layer; (2) embedding layer: each word is mapped to a low-dimensional vector. Given a sentence consisting of T words: s ═ x1，x2，……，xTIs given by the formula ei＝WwrdviEvery word xiConverted into corresponding word vectors eiWherein W iswrdIs a matrix obtained by learning, viIs a vector taking the total amount of words as a dimension; (3) LSTM layer: obtaining high-level features from the embedding layer using a two-way long-short term memory network, wherein the model uses a sum-by-element approach to combine the forward and backward passed outputs; (4) attention layer: and generating a weight vector w, multiplying the word-level feature of each time step by the weight vector, and combining into a sentence-level feature vector. The resulting statement representation for classification: h is*Tanh (r). Wherein r ═ H αT，a＝softmax(wTM), M ═ tanh (H), H is the output vector H ═ H of the LSTM layer1，h2，…，hT](ii) a (5) An output layer: the sentence-level feature vectors are finally used for classification, and the activation function softmax is used to obtain the feature vectors belonging to each device typeAnd probability, wherein the device type with the maximum probability is used as the type of the Internet of things device.
Fig. 4 illustrates the method for extracting the manufacturer information of the internet of things device by using the named entity recognition technology. Namely, the second step specifically comprises:
named entity recognition technology is an entity used to recognize specific meanings in natural language text. The application layer message information becomes a plain text through the step one, and the method identifies the category of each word by using a named entity identification technology. In the step, the two categories are respectively marked as V and O, wherein V represents the category of equipment manufacturers, and O represents other categories.
In the named entity recognition task, the step firstly carries out three different vectorization on the plain text information in the step one, including word vectors, letter vectors and mixed vectors. In the step, letter vector representation of words is carried out by using a gated circulation Unit (GRU) model, finally, word vectors and letter vectors are combined to be used as an independent sequence vector, namely mixed vector representation, and the independent sequence vector is used as input of each gated circulation Unit (GRU) to train a circular neural network model, so that each word in the pure text information in the step one is marked. In the step, a text marked as V is found and used as a manufacturer of the equipment of the Internet of things, namely, a label in the form of (#, equipment manufacturer, #) is provided for the equipment.
Fig. 5 is a flowchart of extracting a product model based on an equipment manufacturer and an existing product information set in the embodiment of the present invention, that is, step three specifically includes:
and setting a window with the length of W in the step based on the equipment manufacturer category V in the step II, finding all the appeared characters in the window, and generating a candidate set B. In this step, an alphabetical level word vector (character embedding) representation and a general word vector (word embedding) representation are performed on each character in the set B. In the step, the known product model name of the internet of things is used as a set A, vector representation of characters in the set B and vector representation of characters in the set A are compared, if the similarity exceeds a threshold value T, the characters (information in the set B, information in the set B and information in the set A, and the similarity exceeds the threshold value T) are used as the product model of the equipment, and the label in the form of (#, #, product model) is provided for the equipment.
Letter-level word vectors and general word vectors are character-level and word-level word vectors. Specifically, the letter-level word vector is obtained by firstly vectorizing letters in a word and then obtaining the vector of the word; the generic word vector is the vector from which the word is directly derived. The former favors the representation of low frequency words and the latter favors the representation of high frequency words.
Claims (4)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110516557.0A CN113191149B (en) | 2021-05-12 | 2021-05-12 | Method for automatically extracting information of Internet of things equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110516557.0A CN113191149B (en) | 2021-05-12 | 2021-05-12 | Method for automatically extracting information of Internet of things equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113191149A true CN113191149A (en) | 2021-07-30 |
| CN113191149B CN113191149B (en) | 2023-04-07 |
Family
ID=76981573
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110516557.0A Active CN113191149B (en) | 2021-05-12 | 2021-05-12 | Method for automatically extracting information of Internet of things equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113191149B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090326923A1 (en) * | 2006-05-15 | 2009-12-31 | Panasonic Corporatioin | Method and apparatus for named entity recognition in natural language |
| CN111726336A (en) * | 2020-05-14 | 2020-09-29 | 北京邮电大学 | A method and system for extracting identification information of a networked intelligent device |
| CN111783466A (en) * | 2020-07-15 | 2020-10-16 | 电子科技大学 | A Named Entity Recognition Method for Chinese Medical Records |
| CN111897962A (en) * | 2020-07-27 | 2020-11-06 | 绿盟科技集团股份有限公司 | Internet of things asset marking method and device |
| CN112564974A (en) * | 2020-12-08 | 2021-03-26 | 武汉大学 | Deep learning-based fingerprint identification method for Internet of things equipment |
-
2021
- 2021-05-12 CN CN202110516557.0A patent/CN113191149B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090326923A1 (en) * | 2006-05-15 | 2009-12-31 | Panasonic Corporatioin | Method and apparatus for named entity recognition in natural language |
| CN111726336A (en) * | 2020-05-14 | 2020-09-29 | 北京邮电大学 | A method and system for extracting identification information of a networked intelligent device |
| CN111783466A (en) * | 2020-07-15 | 2020-10-16 | 电子科技大学 | A Named Entity Recognition Method for Chinese Medical Records |
| CN111897962A (en) * | 2020-07-27 | 2020-11-06 | 绿盟科技集团股份有限公司 | Internet of things asset marking method and device |
| CN112564974A (en) * | 2020-12-08 | 2021-03-26 | 武汉大学 | Deep learning-based fingerprint identification method for Internet of things equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113191149B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111444721B (en) | A method for extracting key information from Chinese text based on pre-trained language model | |
| CN111709241B (en) | Named entity identification method oriented to network security field | |
| CN110851596A (en) | Text classification method and device and computer readable storage medium | |
| CN110019839A (en) | Medical knowledge map construction method and system based on neural network and remote supervisory | |
| CN111950273A (en) | Network public opinion emergency automatic identification method based on emotion information extraction analysis | |
| CN111767725B (en) | Data processing method and device based on emotion polarity analysis model | |
| CN111198948A (en) | Text classification correction method, apparatus, device, and computer-readable storage medium | |
| CN109635288A (en) | A kind of resume abstracting method based on deep neural network | |
| CN113268576B (en) | Deep learning-based department semantic information extraction method and device | |
| CN108388554A (en) | Text emotion identifying system based on collaborative filtering attention mechanism | |
| Sheshikala et al. | Natural language processing and machine learning classifier used for detecting the author of the sentence | |
| CN113204975A (en) | Sensitive character wind identification method based on remote supervision | |
| CN115859980A (en) | Semi-supervised named entity identification method, system and electronic equipment | |
| CN116340506A (en) | A Text Classification Method Based on BERT and Poolless Convolutional Neural Network | |
| CN106874397B (en) | An automatic semantic annotation method for IoT devices | |
| CN115203406A (en) | RoBERTA model-based long text information ground detection method | |
| CN109947936B (en) | A method for dynamic detection of spam based on machine learning | |
| CN115064154A (en) | Method and device for generating mixed language voice recognition model | |
| CN115718792A (en) | Sensitive information extraction method based on natural semantic processing and deep learning | |
| CN114648029A (en) | A Named Entity Recognition Method in Electric Power Field Based on BiLSTM-CRF Model | |
| CN107832307B (en) | Chinese word segmentation method based on undirected graph and single-layer neural network | |
| CN113342935A (en) | Semantic recognition method and device, electronic equipment and readable storage medium | |
| CN112445862A (en) | Internet of things equipment data set construction method and device, electronic equipment and storage medium | |
| US11314922B1 (en) | System and method for generating regulatory content requirement descriptions | |
| Basha et al. | Natural language processing: Practical approach |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |