CN116821155A - Network asset data storage and query method, device and storage medium - Google Patents

Abstract

This application discloses a network asset data storage and query method, device and storage medium, which includes: storing a first document including several second documents in a non-relational database; the second document includes time data and time data Corresponding network asset data, the time data is the time when the network asset data is generated and/or modified; establish a query index based on the attribute data and then create an inverted index table to enable the search engine to query several data groups stored in several nodes in parallel . This application realizes automated network asset management and monitoring, improves management efficiency and accuracy, has good horizontal scalability and distributed processing capabilities, can meet the storage and processing needs of large-scale data, and improves the speed and availability of data storage and query. Scalability.

Description

A network asset data storage and query method, device and storage medium

Technical field

The present application relates to the field of network asset data storage technology, and in particular to a network asset data storage and query method, device and storage medium.

Background technique

In the field of network asset data storage and query, there are some existing methods, such as:

Relational database: Traditional relational database uses a two-dimensional table storage structure based on rows and columns, where each row represents a record and each column represents a field. Relational databases use SQL as the query language and support various operations such as relational operations and aggregate functions. Compared with other databases, relational databases have strong advantages in data consistency, data integrity, transaction processing, etc.

Non-relational databases: Non-relational databases usually use different data models such as key-value, column family, and document to store data. Non-relational databases have strong advantages in horizontal expansion and distributed architecture, so they are widely used in big data processing, Internet applications, real-time data processing and other fields.

Index-based storage: Index-based storage can improve query efficiency by establishing an inverted index for text data. In index-based storage, data is divided into documents, each with its own vocabulary and inverted index. Users can quickly find documents by querying the vocabulary and inverted index. This technology is usually used in scenarios such as full-text search and data mining.

Although there are many database solutions for querying and storing network assets, existing processing methods have certain flaws. For relational databases, relational databases have certain limitations in data structure and query methods. They cannot directly express and query complex relationship networks, and cannot efficiently process large amounts of network asset graph data. At the same time, in large-scale distributed environments, there are also bottlenecks in the performance and scalability of relational databases; for general non-relational databases, although non-relational databases have strong advantages in distributed storage and horizontal expansion , but its storage and query capabilities for complex relationship networks are weak, and it is not suitable for network asset graph data processing; while index-based storage can improve query efficiency by establishing inverted indexes, etc., but it is not suitable for large-scale graph data processing There are still certain limitations. In complex network asset storage and query scenarios, the query speed and efficiency of index-based storage are relatively low.

Contents of the invention

This application provides a network asset data storage and query method, device and storage medium, which can meet the storage and fast query of a large amount of complex network asset data.

In the first aspect, this application provides a network asset data storage method, adopting the following technical solution:

A method for storing network asset data, including the following steps:

Store a first document including several second documents in a non-relational database; the second document includes time data and network asset data corresponding to the time data, and the time data is generated for the network asset data and/or the time of modification.

Furthermore, the above-mentioned non-relational database is a graph database.

Further, the above-mentioned network asset data includes organizational data, domain name data, sub-domain name data, IP data, port data and component data, as well as the relationship between different data; the organizational data, domain name data, sub-domain name data, IP data, port Data and component data are stored as nodes in the graph database, relationships between different data are stored as edge vectors in the graph database, and the nodes are connected through the edge vectors.

Further, the above-mentioned nodes and the edge vectors also include attribute data, and the attribute data are attributes of the network asset data.

Further, the above method also includes calculating the popularity value of the network asset data, and prioritizing preprocessing and caching the asset hot data whose popularity value is greater than a preset popularity threshold in the network asset data.

Further, the above method also includes dividing the time data and the network asset data corresponding to the time data into several data groups, and storing the several data groups in different nodes respectively.

In the second aspect, this application also provides a network asset data query method, which applies the above network asset data storage method and includes the following steps:

Establish query index based on attribute data;

An inverted index table is established according to the query index, so that the search engine can query several data groups stored in several nodes in parallel.

In the third aspect, this application also provides a network asset data storage and query device, including:

A storage module, configured to store a first document including several second documents into a non-relational database;

The query module is used to query several data groups stored in several nodes in parallel through the search engine.

In the fourth aspect, this application provides a computer device that adopts the following technical solution:

A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method as described in the first aspect is implemented.

In the fifth aspect, this application provides a computer-readable storage medium, adopting the following technical solution:

A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods in the first aspect.

To sum up, this application includes at least one of the following beneficial technical effects:

This application provides a network asset data storage and query method, device and storage medium, which realizes automated network asset management and monitoring, improves management efficiency and accuracy, has good horizontal scalability and distributed processing capabilities, and can meet the needs of large-scale users. Scale data storage and processing needs and improve the speed and scalability of data storage and query.

Description of the drawings

Figure 1 is a schematic structural diagram of a network asset database in an embodiment of the present invention.

Figure 2 is a schematic diagram of the network asset data structure in the embodiment of the present invention.

Figure 3 is a schematic flowchart of a network asset database data synchronization method in an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

In order to satisfy the storage and quick query of a large amount of complex network asset graph data, the embodiment of the present application provides a network asset data storage and query method, device and storage medium based on the graph database, which can directly model and query complex network asset data. Relationship network improves the efficiency and accuracy of data processing. Graph database is a data management system based on points and edges as basic storage units and designed on the principle of efficient storage and query of graph data. "Points" represent entities and "edges" represent relationships between entities. In a graph database, the relationships between data are as important as the data itself, and they are stored as part of the data. Such an architecture enables the graph database to quickly respond to complex association queries because the relationships between entities have been stored in the database in advance. Graph databases can intuitively visualize relationships and are the best way to store, query, and analyze highly interconnected data. Common graph databases include Neo4j, Galaxybase, GraphDB, etc.

A relational database refers to a database that uses a relational model to organize data. It stores data in the form of rows and columns to facilitate user understanding. The series of rows and columns in a relational database are called tables, and a set of tables constitutes database. Users retrieve data from a database through queries, which are executable codes that limit certain areas of the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. Mainstream relational databases include Oracle, DB2, MySQL, Microsoft SQLServer, Microsoft Access, etc.

As shown in Figure 2, network asset data as a whole is graph data with hierarchical relationships and connection relationships. For example, data such as "Domain name -> Subdomain name -> IP -> Component". Each domain name has multiple subdomain names. , each subdomain name has a corresponding IP, and the host represented by each IP has different components. These domain names, subdomain names and other data can be called network assets. Network assets have hierarchical relationships that can be constructed by themselves. Topological relationship graphs between network assets. These topological relationship graph data are very suitable for storage in graph databases, because graph databases can effectively represent and query this topology structure.

As shown in Figure 1, this embodiment of the present application discloses a network asset data storage method, which includes the following steps:

Store a first document including several second documents in a non-relational database; the second document includes time data and network asset data corresponding to the time data, and the time data is generated for the network asset data and/or modification time, the above non-relational database is a graph database.

In this implementation, since the relational database has certain limitations in data structure and query methods, it cannot directly express and query complex relationship networks, and it cannot efficiently process a large amount of network asset graph data. Therefore, this application provides A network asset data storage method based on a graph database is proposed. First, use the technical solution of this application to collect all network asset data of the enterprise, including domain names, IP addresses, device information, etc., and store these data in the graph database, where , the first document may include several second documents, and the second document may be a type of network asset data that abstracts network asset data into nodes and directed edges, as well as time data when these network asset data are generated or stored. Among them, a type of network asset data can be different versions of a network asset data at different points in time, or various network asset data at one point in time. There is no limit here;

In one embodiment of the present invention, network asset data includes organizational data, domain name data, sub-domain name data, IP data, port data and component data, as well as relationships between different data; the organizational data, domain name data, sub-domain name data, Domain name data, IP data, port data and component data are stored as nodes in the graph database. The relationships between different data are stored as edge vectors in the graph database. The nodes are connected through the edge vectors. The node and edge vectors also include attribute data, which are attributes of the network asset data.

In this implementation, network asset data can be abstracted into nodes and edge vectors. Nodes represent network assets, and edge vectors represent relationships between different network assets, such as domain names to subdomain names, subdomain names to IP addresses, etc. In our graph database storage engine, each node and edge vector has its own attribute list, and any number of attributes can be added to the nodes and edges. Based on these attribute data, when storing a large number of nodes and edges, the graph database storage engine Optimization techniques such as indexing and caching can be used to speed up queries and navigation between node and edge vectors.

In one embodiment of the present application, the above method further includes calculating the popularity value of the network asset data, and calculating the asset thermal data whose popularity value is greater than a preset popularity threshold in the network asset data, and calculating the asset thermal data. Data is preprocessed and cached first.

In this implementation mode, the technical solution of this application first collects all network asset data of the enterprise, including domain names, IP addresses, device information, etc., and stores these data in a graph database. Preprocessing and caching can be used to precompute and cache commonly used query results to quickly respond to user query requests.

In one embodiment of the present application, the above method further includes dividing the time data and the network asset data corresponding to the time data into several data groups, and storing the several data groups in different locations. within the node.

As shown in Figure 3, in this embodiment, in order to support the storage, synchronization, processing and query of large-scale data, network asset data can be deployed in a distributed manner and the network asset data can be stored on multiple nodes, so that the database has High availability, reliability and scalability. In order to ensure data consistency and reliability between multiple databases, the technical solution of this application uses a database synchronization tool. The database synchronization tool uses message queue technology to collect data from the data source. A message queue, and push the data in the message queue to the target database to achieve synchronization of database information on different nodes.

As shown in Figure 1, in order to meet more practical needs, the technical solution of this application also combines the storage capabilities of a relational database to store a small amount of information that does not need to be stored in a graph database, such as user information. This information can be stored and queried using a relational database, and is associated with nodes and edges in the graph database to achieve comprehensive data management and query functions.

The embodiment of the present application also discloses a network asset data query method, which applies the above network asset data storage method and includes the following steps:

Establish query index based on attribute data;

An inverted index table is established according to the query index, so that the search engine can query several data groups stored in several nodes in parallel.

In this implementation, the graph database storage engine used supports efficient node and edge queries, which facilitates users to query specific network assets based on conditions. Specifically, this graph database storage engine uses Lucene-based full-text search indexing technology to query nodes Create an index with the edge's attributes. This indexing technology can segment the attribute values of nodes and edges and create an inverted index table to record the information of the nodes and edges to which each attribute value belongs. By using this inverted index table, the nodes and edges containing the query conditions provided by the user can be quickly located according to the query conditions provided by the user. At the same time, this patent also uses sharded storage and parallel query methods to divide the data into multiple shards and store them on different nodes, and query multiple shards in parallel, thereby improving query efficiency. Preprocessing and caching are also used to pre-calculate and cache commonly used query results to quickly respond to user query requests.

The embodiment of the present application also discloses a network asset data storage and query device, including:

A storage module, configured to store a first document including several second documents into a non-relational database;

The query module is used to query several data groups stored in several nodes in parallel through the search engine.

The network asset data storage and query device in the embodiment of the present application can implement any of the above-mentioned network asset data storage methods and query methods, and the specific working processes of each module in the network asset data storage and query device can refer to the above method embodiments. corresponding process.

In the several embodiments provided in this application, it should be understood that the methods and systems provided can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules may be combined or integrated. to another system, or some features can be ignored, or not implemented.

An embodiment of the present application also discloses a computer device.

Computer equipment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned xx method is implemented.

An embodiment of the present application also discloses a computer-readable storage medium.

A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above xx methods.

Among them, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in combination with an instruction execution system, device or device; the program code contained on the computer-readable medium can be used in any appropriate medium. Transmission, including but not limited to wireless, wires, optical cables, RF, etc., or any suitable combination of the above.

Example:

If a user or enterprise owns a large number of network assets, these assets need to be managed and monitored to ensure the security and stability of the network. The traditional management method requires manual entry and management, which is inefficient and error-prone. The technical solution of this application can be used to realize automated network asset management and monitoring and improve management efficiency and accuracy. The specific steps are as follows:

S101. Data collection: First, use the technical solution of this application to collect all network asset data of the enterprise, including domain names, IP addresses, device information, etc., and store these data in the graph database. Preprocessing and caching can be used to precompute and cache commonly used query results to quickly respond to user query requests.

S102. Data modeling: Abstract the collected network asset data into the form of nodes and edges, and use a graph database to store these nodes and edges as a graph structure. Nodes represent network assets, and edges represent relationships between different network assets, such as domain names to subdomain names, subdomain names to IP addresses, etc. This graphical modeling method can better support the query and relationship analysis of network asset data.

S103. Data management: At the same time, some small amounts of information that do not need to be stored in the graph database, such as user information, are stored in the relational database, and are associated with the nodes and edges in the graph database to achieve comprehensive data management. and query functions.

S104. Data query: Users can quickly query the desired network asset data according to their needs, such as a series of relationship data such as subdomain names related to a certain domain name, IP under the subdomain name, etc.

S105. Data security: Use security strategies such as data encryption and identity verification to ensure data security. At the same time, use the multi-database synchronization function to synchronize data to multiple databases and perform backup and recovery operations, thereby improving data reliability and availability.

Through the above processing steps, automated network asset management and monitoring can be achieved, improving management efficiency and accuracy.

It can be seen from the above that the network asset data storage and query method, device and storage medium provided by this application use a graph database as the core storage engine on the one hand, abstract the network asset data into nodes and edges, and store it in a graph structure, so that it can Perform queries and relationship analysis more quickly, and better support graphical modeling and query of network asset data; on the other hand, full-text search indexing technology based on Lucene is used to index the attributes of nodes and edges to facilitate user queries. The nodes and edges containing these conditions can be quickly located based on the query conditions provided by the user. The method of sharded storage and parallel query is adopted to store and query large-scale network asset data, which can improve query efficiency. At the same time, pre-processing and caching are used to pre-calculate and cache commonly used query results to quickly respond to user query requests. Improve the efficiency of storage and query; in addition to using the graph database to store nodes and edges, it also combines the storage capabilities of the relational database to store a small amount of information that does not need to be stored in the graph database. This information can be stored and queried using a relational database, and is associated with nodes and edges in the graph database to achieve comprehensive data management and query functions; the above graph databases and relational databases use multi-database synchronization functions , you can synchronize data to multiple databases and perform backup and recovery operations, thereby improving data reliability and availability. Use security strategies such as data encryption and authentication to ensure data security and improve data security and availability.

It should be noted that in the above-mentioned embodiments, each embodiment is described with different emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Any feature disclosed in this specification (including the abstract and the drawings), unless specifically stated, can be replaced by other equivalent or similar features. Replace it with the alternative characteristics of the purpose. That is, unless otherwise stated, each feature is only one example in a series of equivalent or similar features.

Claims (10)

1. A method of storing network asset data, comprising the steps of:

storing a first document comprising a plurality of second documents in a non-relational database; the second document includes time data and network asset data corresponding to the time data, the time data being a time at which the network asset data was generated and/or modified.

2. The network asset data storage method of claim 1, wherein: the non-relational database is a graph database.

3. The network asset data storage method of claim 2, wherein: the network asset data includes organization data, domain name data, sub domain name data, IP data, port data, and relationships between component data; the organization data, the domain name data, the sub domain name data, the IP data, the port data and the component data are stored as nodes in the graph database, the relation between different data is stored as an edge vector in the graph database, and the nodes are connected through the edge vector.

4. A method of storing network asset data according to claim 3, characterized by: the nodes and the edge vectors further include attribute data, which is an attribute of the network asset data.

5. A network asset data storage method according to claim 1 or 3, characterized in that: and calculating the heat value of the network asset data, preprocessing and caching the asset heat data with priority, wherein the heat value of the asset heat data is larger than a preset heat threshold value.

6. The network asset data storage method of claim 5, further comprising dividing the time data and the network asset data corresponding to the time data into a plurality of data groups, and storing the plurality of data groups in different nodes, respectively.

7. A network asset data query method, applying a network asset data storage method according to any one of claims 1-6, comprising the steps of:

establishing a query index according to the attribute data;

and establishing an inverted index table according to the query index so that the search engine queries a plurality of data sets stored in a plurality of nodes in parallel.

8. A network asset data storage and querying device, comprising:

the storage module is used for storing a first document comprising a plurality of second documents into the non-relational database;

and the query module is used for querying a plurality of data sets stored in a plurality of nodes in parallel through a search engine.

9. A computing device, characterized by: comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method according to any of claims 1 to 7 when the program is executed.

10. A computer-readable storage medium, characterized by: a computer program being stored which can be loaded by a processor and which performs the method according to any one of claims 1 to 7.