CN113626509A - Data access method and device, electronic equipment and readable storage medium - Google Patents
Data access method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN113626509A CN113626509A CN202110909175.4A CN202110909175A CN113626509A CN 113626509 A CN113626509 A CN 113626509A CN 202110909175 A CN202110909175 A CN 202110909175A CN 113626509 A CN113626509 A CN 113626509A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- attack
- query
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000003860 storage Methods 0.000 title claims abstract description 30
- 238000004140 cleaning Methods 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 29
- 230000002452 interceptive effect Effects 0.000 claims description 22
- 230000008569 process Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 230000002776 aggregation Effects 0.000 abstract description 8
- 238000004220 aggregation Methods 0.000 abstract description 8
- 230000009286 beneficial effect Effects 0.000 abstract description 8
- 230000006870 function Effects 0.000 description 13
- 238000012795 verification Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000013500 data storage Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000004873 anchoring Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 231100000279 safety data Toxicity 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000010926 purge Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Abstract
The application discloses a data access method, a data access device, electronic equipment and a readable storage medium. The method comprises the steps of when a data pushing request is received, placing received target data into a corresponding target position according to client information and a data type corresponding to the data pushing request. Calling a corresponding target engine to read the target data from the target position according to the data source of the target data; if the target data is not historical attack data, filling attack attribute information for the target data to obtain safe target data; and formatting and cleaning the safety target data according to the data source, and uniformly mapping the safety target data to the data dictionary corresponding to the belonged category. The method and the device can realize standardized access aiming at different structured data, are beneficial to simply and efficiently realizing data query, are beneficial to realizing multi-dimensional statistics and aggregation analysis of data, and can effectively support the service side function.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data access method, an apparatus, an electronic device, and a readable storage medium.
Background
Currently, in the field of big data technologies, data access systems are in a wide range, and many existing data access systems access data of various manufacturers in real time in a message queue manner to support service platforms, for example, data manufacturers accessing to the green alliance, the deep belief, the odd security letter and the like. The related technology receives the message sent by the transaction center through a message receiving component; the received message is analyzed through the message analyzing component to obtain target data, and the obtained target data is stored through the data storage component, so that data access is realized.
However, because the data formats of manufacturers are not completely the same, and the data storage formats in storage are not uniform, different service layer interfaces need to be encapsulated for data of different formats in the aspect of service side use, which increases the difficulty in implementing data query; subsequent multi-dimensional statistics and aggregate analysis become extremely difficult, and complex business scenes cannot be effectively supported.
Disclosure of Invention
The application provides a data access method, a data access device, an electronic device and a readable storage medium, which can realize standardized access aiming at different structured data, are beneficial to simply and efficiently realizing data query, are beneficial to realizing multi-dimensional statistics and aggregation analysis of data, and can effectively support the function of a business side.
In order to solve the above technical problems, embodiments of the present invention provide the following technical solutions:
an embodiment of the present invention provides a data access method, including:
when a data pushing request is received, placing the received target data into a corresponding target position according to client information and a data type corresponding to the data pushing request;
calling a corresponding target engine to read the target data from the target position according to the data source of the target data;
if the target data is not historical attack data, filling attack attribute information for the target data to obtain safe target data;
and carrying out formatting cleaning on the safety target data according to the data source, and uniformly mapping the safety target data to a data dictionary corresponding to the category to which the safety target data belongs.
Optionally, after receiving the data pushing request, the method further includes:
distributing access permission identification information for an authorized interactive terminal in advance;
judging whether the data pushing request carries the access permission identification information or not;
if the data pushing request does not carry the access permission identification information, determining that the client corresponding to the data pushing request is not an authorized interactive end;
if the data pushing request carries the access permission identification information, the client corresponding to the data pushing request is judged to be an authorized interactive end, and meanwhile, the step of placing the received target data into the corresponding target position according to the client information and the data type corresponding to the data pushing request is executed.
Optionally, after allocating the access permission identification information to the authorized interactive terminal, the method further includes:
when a data query instruction is received, judging whether the data query request carries the access permission identification information;
if the data pushing request does not carry the access permission identification information, determining that the client corresponding to the data query request is not an authorized interactive end;
if the data query request carries the access permission identification information, determining that the client corresponding to the data push request is an authorized interactive end, and simultaneously responding to the data query instruction.
Optionally, the process of responding to the data query instruction includes:
judging whether the data query type corresponding to the data query instruction is a basic query type or a multiple query type;
if the data query type corresponding to the data query instruction is the basic query type, calling a basic API (application program interface) to query a corresponding database, and feeding back a data query result;
if the data query type corresponding to the data query instruction is the multiple query type, calling a basic API interface for multiple times to query the corresponding database, and obtaining a final data query result according to the query result of each database.
Optionally, the process of filling attack attribute information for the target data to obtain secure target data includes:
an attack database is configured in advance, the attack database comprises a plurality of pieces of historical attack data, and each piece of historical attack data carries unit information of a corresponding attack domain name and system website information;
automatically analyzing geographical position information according to the IP address of the data pushing request, and filling the geographical position information serving as attack attribute information into the target data;
based on the attack database, according to the uniform resource locator of the data push request, unit information of the domain name and corresponding system website information are automatically filled in the target data.
Optionally, before the target data is not historical attack data, the method further includes:
acquiring a unique identifier of the target data, an attack source IP and an attack target IP;
judging whether the target data is historical attack data of the attack database or not according to the unique identifier, the attack source IP and the attack target IP;
if the target data is historical attack data of the attack database, discarding the target data;
and if the target data is not the historical attack data of the attack database, judging that the target data is not the historical attack data.
Optionally, after the unified mapping is performed on the data dictionary corresponding to the category to which the unified mapping belongs, the method further includes:
creating a corresponding database group for each engine in advance, wherein each database group is provided with a plurality of sub-libraries according to the data type;
matching a target database group according to the data source of the target data;
and matching corresponding target sub-libraries in the target database group according to the data types of the target data, and storing the mapped target data to the target sub-libraries.
Another aspect of the embodiments of the present invention provides a data access apparatus, including:
the data receiving module is used for placing the received target data into a corresponding target position according to the client information and the data type corresponding to the data pushing request when the data pushing request is received;
the data reading module is used for calling a corresponding target engine to read the target data from the target position according to the data source of the target data;
the data processing module is used for filling attack attribute information for the target data to obtain safe target data if the target data is not historical attack data;
and the standardization module is used for carrying out formatting cleaning on the safety target data according to the data source and uniformly mapping the safety target data to the data dictionary corresponding to the category to which the safety target data belongs.
An embodiment of the present invention further provides an electronic device, which includes a processor, and the processor is configured to implement the steps of the data access method according to any one of the foregoing items when executing the computer program stored in the memory.
Finally, an embodiment of the present invention provides a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the data access method according to any of the foregoing items.
The technical scheme provided by the application has the advantages that received original data with different structuralization is placed in a message queue, different data processing engines process the data, and attack data discrimination, attribute filling and formatting cleaning operations are carried out on the original data, so that standardized access to the different structuralization data is realized; the data to be stored is standardized data, different service layer interfaces are not required to be packaged, the data query difficulty is simplified, the data query is easily, conveniently and efficiently realized, the multi-dimensional statistics and aggregation analysis of the data are further realized, and the service side function can be effectively supported.
In addition, the embodiment of the invention also provides a corresponding implementation device, electronic equipment and a readable storage medium for the data access method, so that the method has higher practicability, and the device, the electronic equipment and the readable storage medium have corresponding advantages.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the related art, the drawings required to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data access method according to an embodiment of the present invention;
FIG. 2 is a block diagram of an exemplary application scenario provided by an embodiment of the present invention;
fig. 3 is a flowchart of a data access method according to an exemplary embodiment of the present invention;
FIG. 4 is a flowchart illustrating a data query method according to an exemplary embodiment of the present invention;
fig. 5 is a structural diagram of a specific embodiment of a data access device according to an embodiment of the present invention;
fig. 6 is a block diagram of an embodiment of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may include other steps or elements not expressly listed.
Having described the technical solutions of the embodiments of the present invention, various non-limiting embodiments of the present application are described in detail below.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data access method according to an embodiment of the present invention, where the embodiment of the present invention may include the following:
s101: and when a data pushing request is received, putting the received target data into a corresponding target position according to the client information and the data type corresponding to the data pushing request.
The data push request of this embodiment is a request sent by a data manufacturer to a data access system, and is used to insert data of the data manufacturer into the data access system, where the data access system is an execution subject of this embodiment. The client information is also the basic information of the data manufacturer, such as name, data format, etc. The data type may be, for example, attack data, a security event, a vulnerability, and the like, and the target data to be put into the data access system corresponding to the data push request is first input into the matched message queue according to the data vendor and the data type. In this embodiment, a plurality of input queues or message queues are constructed in advance according to different data manufacturers, one input queue corresponds to one data manufacturer, for each input queue, a plurality of storage nodes can be constructed in the input queue according to different data types, and each node stores data of one data type. As an alternative implementation, the embodiment may use the kafka cluster for data storage, and place the accessed target data into the corresponding Topic according to different data vendors and data types. Because the kafka cluster allows a single kafka node in the cluster to fail, and supports thousands of clients to read and write hot expansion simultaneously, the fault tolerance, high concurrency and expandability of a data access system can be improved. In addition, for the data access system, the data access system can simultaneously receive a client and simultaneously send a plurality of data push requests, a plurality of clients can also simultaneously send a plurality of data push requests, and certainly, only one data push request is received at the same time, and technicians in the field can flexibly select the data access system according to actual requirements.
S102: and calling a corresponding target engine to read the target data from the target position according to the data source of the target data.
The data source of this embodiment refers to which client or data of the data vendor the data source is, for example, the data source of the target data may be a secure data source, that is, the target data is a secure data source. The data source is used for deciding which data manufacturer' S engine to call, and the engine is used for reading all data from the target position in sequence, and executing the steps S103 and S104 to eliminate the structural difference of different data sources. When the engine reads data from the target position, the data can be read at a fixed frequency, for example, every 10s, or in real time, or the data can be read when the target position is monitored, which does not affect the implementation of the present application.
S103: and if the target data is not historical attack data, filling attack attribute information for the target data to obtain safe target data.
It will be appreciated that each piece of attack data may be uniquely identified by a unique identification ID, attack source IP and attack target IP fields. A file or a database for storing historical attack data can be constructed in advance, and after target data are obtained, duplicate removal of the attack data is carried out according to the unique identification ID of the target data, the attack source IP and the attack target IP field. The attack attribute information includes, but is not limited to, geographical location information, information of a unit where a domain name is located, and system website information, and those skilled in the art can flexibly add or delete the attack attribute information according to actual needs, which does not affect the implementation of the present application.
S104: and formatting and cleaning the safety target data according to the data source, and uniformly mapping the safety target data to the data dictionary corresponding to the belonged category.
Similarly, the data source is used to determine the called engine, and the engine corresponding to the data source is used to perform the formatting and cleaning operation on the data obtained in the previous step. Each type of data in the data access system is provided with a group of data dictionaries, and the same type of data of different data sources can be uniformly mapped into the data dictionaries after field formatting cleaning and stored in corresponding databases, so that the development amount and difficulty of service modules are greatly simplified.
In the technical scheme provided by the embodiment of the invention, the received original data with different structures are put into a message queue, different data processing engines process the data, and the original data is subjected to attack data discrimination, attribute filling and formatting cleaning operation, so that the standardized access to the different structured data is realized; the data to be stored is standardized data, different service layer interfaces are not required to be packaged, the data query difficulty is simplified, the data query is easily, conveniently and efficiently realized, the multi-dimensional statistics and aggregation analysis of the data are further realized, and the service side function can be effectively supported.
It can be understood that, in the related art, data is received through the message component, that is, the data of each manufacturer is simply accessed into the system, no limitation is imposed on the accessed data source manufacturer, an unknown manufacturer can also push data into the message component of the system in the past, and the system is likely to be attacked maliciously by the unknown manufacturer, and the security is low. In order to solve the technical disadvantage, based on the above embodiment, the present application further sets an authority verification function for the data access system, verifies the client of the data push request, and determines whether the client is a data vendor that can allow access, and the implementation process is as follows:
and allocating access permission identification information to the authorized interactive terminal in advance. Judging whether the data pushing request carries access permission identification information or not; if the data pushing request does not carry access permission identification information, judging that the client corresponding to the data pushing request is not an authorized interactive end; if the data pushing request carries the access permission identification information, the client corresponding to the data pushing request is judged to be an authorized interactive end, and the step S101 is executed at the same time.
The access permission identification information is used as a criterion of the data access system for determining whether the client is authorized to access the data access system, and may be, for example, a token. The client can register in the data access system in advance, apply for the data access authority after registering successfully, the data access system agrees to the data manufacturer to insert, will distribute the access permission identification information for it, while carrying the access permission identification information when the manufacturer and data access system carry on the data interaction subsequently, in order to prove it is the identity of the authorized user. In a specific implementation, the permission authentication computer program may be encapsulated in an interface layer of the data access system, that is, the computer program that this embodiment relies on is implemented, and after receiving an external request, all interfaces of the interface layer need to call the permission authentication computer program to perform permission authentication.
The embodiment encapsulates the data access API, embeds the authority authentication, uniformly manages the external authority of the interface, and improves the security of data access by limiting a data source manufacturer to transmit data to the system.
It can be understood that the data query function is a basic function of the data access system, and based on the above embodiment, this embodiment further provides an implementation flow of the data query function, which may include the following contents:
when a data query instruction is received, judging whether the data query type corresponding to the data query instruction is a basic query type or a multiple query type; if the data query type corresponding to the data query instruction is the basic query type, calling a basic API interface to query a corresponding database, and feeding back a data query result; if the data query type corresponding to the data query instruction is a multiple query type, calling the basic API interface for multiple times to query the corresponding database, and obtaining a final data query result according to the query result of each database.
In this embodiment, the basic query type is a data query function that can implement service logic by directly requesting a database, such as basic accurate query, fuzzy query, and packet query implemented at a service level. The multiple query type is that business logic can be realized through multiple basic query requests, such as distribution statistics of multiple dimensions, time axis statistics, importance aggregation statistics and the like.
As an optional implementation manner of this embodiment, in order to further improve the security of the data access system, the present application further needs to perform authority verification on the data query instruction, which may include the following contents:
when a data query instruction is received, judging whether the data query request carries access permission identification information or not; if the data pushing request does not carry access permission identification information, judging that the client corresponding to the data query request is not an authorized interactive end; and if the data query request carries the access permission identification information, judging that the client corresponding to the data push request is an authorized interactive terminal, and responding to the data query instruction.
In the foregoing embodiment, how to execute step S103 is not limited, and an implementation manner of adding attack attribute information in this embodiment may include the following steps:
an attack database is configured in advance, the attack database comprises a plurality of pieces of historical attack data, and each piece of historical attack data carries unit information corresponding to an attack domain name and system website information of the historical attack data. And automatically analyzing the geographical position information according to the IP address of the data push request, for example, automatically analyzing the province, city and district information of the IP according to the IP address, and filling the geographical position information serving as the attack attribute information in the target data. Based on the attack database, according to the uniform resource locator URL of the data push request, unit information where the domain name is located and corresponding system website information are automatically filled into the target data. Accordingly, the embodiment of determining whether the target data is historical attack data may be: acquiring a unique identifier of target data, an attack source IP and an attack target IP; judging whether the target data is historical attack data of an attack database or not according to the unique identifier, the attack source IP and the attack target IP; and if the target data is historical attack data of the attack database, discarding the target data, and reading next data from the target position as the currently processed target data. And if the target data is not the historical attack data of the historical attack database, judging that the target data is not the historical attack data.
By configuring the database in advance, the efficiency of data deduplication operation and attribute filling operation is improved.
In order to facilitate multi-dimensional statistical and aggregate analysis on the standardized data, based on the above embodiments, the present application further provides a corresponding storage manner, which may include the following:
and creating a corresponding database group for each engine in advance, wherein each database group is provided with a plurality of sub-libraries according to the data type. For example, the data processed by the security engine is stored in a database group with the manufacturer suffix of anchoring, and the database group can be subdivided into an attack library, a security event library, a vulnerability library and the like according to the data type. The target database group can be matched according to the data source of the target data, then the corresponding target sub-base is matched in the target database group according to the data type of the target data, and the mapped target data is stored in the target sub-base.
In order to make the technical solutions of the present application more obvious to those skilled in the art, the present application also provides an illustrative example in conjunction with fig. 2 to 4, which may include the following:
the data access architecture constructed by the embodiment includes an interface layer, a service layer, a middle layer, an engine layer and a storage layer, as shown in fig. 2. The interface layer is used for data interaction of the system to the outside, and manages the external authority in a unified manner, for example, the authority authentication can be realized by Spring Security + JWT. For a platform or a manufacturer with data interaction with the system, a special account is set for the platform or the manufacturer, an authority verification interface is provided, and whether a user name and a password are correct or not is verified. If the verification is successful, the token is returned as the access permission identification information, and the subsequent third party manufacturer needs to bring the token to access the interface of the system to have access right. The interface layer implementation can be divided into two major categories, namely a query API which is only responsible for data query and a data API for data access and data update. The type of the query API is refined inside the service layer: 1) the basic API of the service logic can be realized by directly requesting the database, such as basic accurate query, fuzzy query, grouping query and the like are realized at the service level; 2) the high-level API of the business logic, namely the multiple API, can be realized through multiple basic API requests, such as the distribution statistics, the time axis statistics, the importance aggregation statistics and the like of multiple dimensions. The middle layer may be implemented by kafka, and is dedicated to the data API to insert a data input queue for processing by the data wait engine layer. Data are put into different topics, so that different data engines can process the data conveniently. For example, when the data API receives data from a security vendor, a corresponding field may identify the data type to which the data belongs, and then place data of different data types into corresponding Topic. The engine layer, as a data processing layer, may include a plurality of engines that perform data processing operations, each data processing engine being partitioned based on a different data source. If the data source is a steady data source, the corresponding steady data processing engine is used for processing the data, and data de-duplication, attribute filling, formatting cleaning and finally standardized warehousing are carried out on the data. The storage layer is an elastic search-based storage layer, which performs grouping processing on the databases, and each engine corresponds to one database group. For example, the data processed by the security engine is stored in a database group with the manufacturer suffix of anchoring, and the database group can be subdivided into an attack library, a security event library, a vulnerability library and the like.
The following describes a data processing flow by taking an engine as an immobilizer engine as an example, and may include an original data pull operation, a data deduplication operation, an attribute information fill operation, and a formatting purge operation. Pulling original data: the immobilizer engine constantly polls to pull raw data from the corresponding topic. The number of data pieces pulled in a single time and the data pulling interval time can be flexibly configured according to the data volume and the performance of the server, and the application does not limit the data pieces in the single time and the data pulling interval time. The default is to pull 1000 strips in a single time, once every 1 second. Data pull operations between multiple engines may be performed asynchronously. Data deduplication: each piece of attack data necessarily comprises a unique identification ID, an attack source IP and an attack target IP field, and all the attack data sent by manufacturers can determine uniqueness by using the three fields, so that the attack data is deduplicated. And when the unique identification ID, the attack source IP and the attack target IP field of the current pulled data are all repeated with one data in the database, directly skipping the processing logic and entering the processing flow of the next data. And (3) attribute filling: the original data generally contains information such as attack source IP, attack target IP, attacked domain name and the like, and the information is not very intuitive. In the process of processing data by the engine layer, the province, city and district information of the IP is automatically analyzed according to the IP address, and the geographical position information of the original data is filled; and for the attacked domain name, automatically filling unit information of the domain name and system website information according to the uniform resource locator URL. Formatting and cleaning: and formatting and cleaning the data extracted from the Topic based on the format requirement of the safety data source, and uniformly mapping the data to a data dictionary of the safety data source.
The data access system based on the above architecture, with reference to fig. 3, describes a data access process by taking an IP domain name library as an example: the data API of the interface layer is responsible for receiving data actively pushed by a client, such as an agility data manufacturer, and receives the latest IP domain name data pushed by a plurality of clients, and the specific process is as follows: the client side initiates an authentication request to the data API of the interface layer, the data API returns an authentication pass, and the client side sends an IP domain name data list to the data API according to the authentication information. And the data API puts the received IP domain name data list into a data input queue of the middle layer and simultaneously returns feedback information of success of the newly added data request to the client. The IP domain name engine focuses on the IP domain name information Topic of the middle layer and pulls data from the Topic at intervals. And the IP domain name engine performs data deduplication processing, attribute filling processing and formatting cleaning processing on the pulled IP domain name data, and stores the finally obtained data into an IP domain name library of a storage layer to finish the whole data access process.
The query is the most basic requirement of a system, and the data access system based on the above architecture explains the flow of querying data by taking querying IP domain name data as an example in combination with fig. 4: the client side initiates an authentication request to the query API of the interface layer, and the query API returns that the authentication is passed. The client sends a specific request for inquiring IP domain name data to the inquiry API according to the authentication information, and the inquiry API judges that the request belongs to the basic API and forwards the request to the basic API. And inquiring the IP domain name data from the IP domain name library by the basic API, and acquiring a result from the IP domain name library by the basic API. After the data query result fed back by the IP domain name library is obtained, the basic API returns the query result to the query API, and the query API returns the query result to the client.
As can be seen from the above, in the embodiment, data is accessed through the API interface, and the external authority of the interface is uniformly managed through the embedded authority authentication of the interface, so that the related authority of a data source manufacturer that transmits data to the system can be effectively controlled, and the security of data access is improved. And thirdly, the system performs data deduplication, attribute filling and formatting cleaning on the accessed data, eliminates the structural difference of different data sources, realizes standardized access on different structural data and simplifies the development of business logic.
It should be noted that, in the present application, there is no strict sequential execution order among the steps, and as long as a logical order is met, the steps may be executed simultaneously or according to a certain preset order, and fig. 1, fig. 3, and fig. 4 are only schematic manners, and do not represent that only such an execution order is available.
The embodiment of the invention also provides a corresponding device for the data access method, thereby further ensuring that the method has higher practicability. Wherein the means can be described separately from the functional module point of view and the hardware point of view. In the following, the data access apparatus provided by the embodiment of the present invention is introduced, and the data access apparatus described below and the data access method described above may be referred to correspondingly.
Based on the angle of the functional module, referring to fig. 5, fig. 5 is a structural diagram of a data access device according to an embodiment of the present invention, in a specific implementation manner, where the data access device may include:
the data receiving module 501 is configured to, when receiving a data pushing request, place the received target data into a corresponding target location according to client information and a data type corresponding to the data pushing request.
The data reading module 502 is configured to invoke a corresponding target engine to read the target data from the target location according to the data source of the target data.
And the data processing module 503 is configured to, if the target data is not historical attack data, fill attack attribute information for the target data to obtain secure target data.
And the standardization module 504 is used for performing formatting cleaning on the safety target data according to the data source and uniformly mapping the safety target data to the data dictionary corresponding to the category to which the safety target data belongs.
Optionally, in some embodiments of this embodiment, the apparatus may further include an authority verification module, configured to pre-allocate access permission identification information to the authorized interactive terminal; judging whether the data pushing request carries access permission identification information or not; if the data pushing request does not carry access permission identification information, judging that the client corresponding to the data pushing request is not an authorized interactive end; and if the data pushing request carries the access permission identification information, judging that the client corresponding to the data pushing request is an authorized interactive end.
As an optional implementation manner of this embodiment, the apparatus may further include a data query module, configured to determine, when the data query instruction is received, whether the data query type corresponding to the data query instruction is a basic query type or a multiple query type; if the data query type corresponding to the data query instruction is the basic query type, calling a basic API interface to query a corresponding database, and feeding back a data query result; if the data query type corresponding to the data query instruction is a multiple query type, calling the basic API interface for multiple times to query the corresponding database, and obtaining a final data query result according to the query result of each database.
As another optional implementation manner of this embodiment, the right verification module may further be configured to: when a data query instruction is received, judging whether the data query request carries access permission identification information or not; if the data pushing request does not carry access permission identification information, judging that the client corresponding to the data query request is not an authorized interactive end; and if the data query request carries the access permission identification information, judging that the client corresponding to the data push request is an authorized interactive terminal, and responding to the data query instruction.
Optionally, in other embodiments of this embodiment, the data processing module 503 may be further configured to: an attack database is configured in advance, the attack database comprises a plurality of pieces of historical attack data, and the historical attack data carry unit information of corresponding attack domain names and system website information; automatically analyzing the geographical position information according to the IP address of the data pushing request, and filling the geographical position information serving as attack attribute information into target data; based on the attack database, unit information of the domain name and corresponding system website information are automatically filled into the target data according to the uniform resource locator of the data push request.
As an optional implementation manner of this embodiment, the data processing module 503 may further include a deduplication unit, configured to obtain a unique identifier of the target data, an attack source IP, and an attack target IP; judging whether the target data is historical attack data of an attack database or not according to the unique identifier, the attack source IP and the attack target IP; if the target data is historical attack data of the attack database, discarding the target data; and if the target data is not the historical attack data of the historical attack database, judging that the target data is not the historical attack data.
Optionally, in some other embodiments of this embodiment, the apparatus may further include a data storage module, for example, configured to create a corresponding database group for each engine in advance, where each database group sets a plurality of sub-libraries according to data types; matching the target database group according to the data source of the target data; and matching corresponding target sub-libraries in the target database group according to the data types of the target data, and storing the mapped target data to the target sub-libraries.
The functions of each functional module of the data access device in the embodiments of the present invention may be specifically implemented according to the method in the above method embodiments, and the specific implementation process may refer to the related description of the above method embodiments, which is not described herein again.
Therefore, the embodiment of the invention can realize standardized access aiming at different structured data, is beneficial to simply and efficiently realizing data query, is beneficial to realizing multi-dimensional statistics and aggregation analysis of data, and can effectively support the service side function.
The above mentioned data access device is described from the perspective of a functional module, and further, the present application also provides an electronic device, which is described from the perspective of hardware. Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 6, the electronic device includes a memory 60 for storing a computer program; a processor 61, configured to execute a computer program to implement the steps of the data access method according to any of the above embodiments.
The processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 61 may also be a controller, a microcontroller, a microprocessor or other data processing chip, and the like. The processor 61 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that the display screen needs to display. In some embodiments, the processor 61 may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.
Memory 60 may include one or more computer-readable storage media, which may be non-transitory. Memory 60 may also include high speed random access memory as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. The memory 60 may in some embodiments be an internal storage unit of the electronic device, for example a hard disk of a server. The memory 60 may also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk provided on a server, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the memory 60 may also include both internal storage units of the electronic device and external storage devices. The memory 60 may be used for storing various data and application software installed in the electronic device, such as: the code of the program that executes the vulnerability handling method, etc. may also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 60 is at least used for storing a computer program 601, wherein the computer program is loaded and executed by the processor 61, and then the relevant steps of the data access method disclosed in any of the foregoing embodiments can be implemented. In addition, the resources stored by the memory 60 may also include an operating system 602, data 603, and the like, and the storage may be transient storage or permanent storage. Operating system 602 may include Windows, Unix, Linux, etc., among others. The data 603 may include, but is not limited to, data corresponding to the data access result, and the like.
In some embodiments, the electronic device may further include a display 62, an input/output interface 63, a communication interface 64, otherwise known as a network interface, a power supply 65, and a communication bus 66. The display 62 and the input/output interface 63, such as a Keyboard (Keyboard), belong to a user interface, and the optional user interface may also include a standard wired interface, a wireless interface, and the like. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, as appropriate, is used for displaying information processed in the electronic device and for displaying a visualized user interface. The communication interface 64 may optionally include a wired interface and/or a wireless interface, such as a WI-FI interface, a bluetooth interface, etc., typically used to establish a communication link between an electronic device and other electronic devices. The communication bus 66 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
Those skilled in the art will appreciate that the configuration shown in fig. 6 is not intended to be limiting of the electronic device and may include more or fewer components than those shown, such as a sensor 67 that performs various functions.
The functions of the functional modules of the electronic device according to the embodiments of the present invention may be specifically implemented according to the method in the above method embodiments, and the specific implementation process may refer to the description related to the above method embodiments, which is not described herein again.
Therefore, the embodiment of the invention can realize standardized access aiming at different structured data, is beneficial to simply and efficiently realizing data query, is beneficial to realizing multi-dimensional statistics and aggregation analysis of data, and can effectively support the service side function.
It is to be understood that, if the data access method in the above embodiments is implemented in the form of software functional units and sold or used as a stand-alone product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application may be substantially or partially implemented in the form of a software product, which is stored in a storage medium and executes all or part of the steps of the methods of the embodiments of the present application, or all or part of the technical solutions. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a multimedia card, a card type Memory (e.g., SD or DX Memory, etc.), a magnetic Memory, a removable magnetic disk, a CD-ROM, a magnetic or optical disk, and other various media capable of storing program codes.
Based on this, the embodiment of the present invention further provides a readable storage medium, which stores a computer program, and the computer program is executed by a processor, and the steps of the data access method according to any one of the above embodiments are provided.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. For hardware including devices and electronic equipment disclosed by the embodiment, the description is relatively simple because the hardware includes the devices and the electronic equipment correspond to the method disclosed by the embodiment, and the relevant points can be obtained by referring to the description of the method.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The data access method, the data access device, the electronic device, and the readable storage medium provided by the present application are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A data access method, comprising:
when a data pushing request is received, placing the received target data into a corresponding target position according to client information and a data type corresponding to the data pushing request;
calling a corresponding target engine to read the target data from the target position according to the data source of the target data;
if the target data is not historical attack data, filling attack attribute information for the target data to obtain safe target data;
and carrying out formatting cleaning on the safety target data according to the data source, and uniformly mapping the safety target data to a data dictionary corresponding to the category to which the safety target data belongs.
2. The data access method according to claim 1, wherein after receiving the data push request, further comprising:
distributing access permission identification information for an authorized interactive terminal in advance;
judging whether the data pushing request carries the access permission identification information or not;
if the data pushing request does not carry the access permission identification information, determining that the client corresponding to the data pushing request is not an authorized interactive end;
if the data pushing request carries the access permission identification information, the client corresponding to the data pushing request is judged to be an authorized interactive end, and meanwhile, the step of placing the received target data into the corresponding target position according to the client information and the data type corresponding to the data pushing request is executed.
3. The data access method according to claim 2, wherein after the allocating the access permission identification information to the authorized interactive terminal, the method further comprises:
when a data query instruction is received, judging whether the data query request carries the access permission identification information;
if the data pushing request does not carry the access permission identification information, determining that the client corresponding to the data query request is not an authorized interactive end;
if the data query request carries the access permission identification information, determining that the client corresponding to the data push request is an authorized interactive end, and simultaneously responding to the data query instruction.
4. The data access method of claim 3, wherein the process of responding to the data query instruction comprises:
judging whether the data query type corresponding to the data query instruction is a basic query type or a multiple query type;
if the data query type corresponding to the data query instruction is the basic query type, calling a basic API (application program interface) to query a corresponding database, and feeding back a data query result;
if the data query type corresponding to the data query instruction is the multiple query type, calling a basic API interface for multiple times to query the corresponding database, and obtaining a final data query result according to the query result of each database.
5. The data access method according to any one of claims 1 to 4, wherein the process of filling attack attribute information for the target data to obtain secure target data includes:
an attack database is configured in advance, the attack database comprises a plurality of pieces of historical attack data, and each piece of historical attack data carries unit information of a corresponding attack domain name and system website information;
automatically analyzing geographical position information according to the IP address of the data pushing request, and filling the geographical position information serving as the attack attribute information in the target data;
based on the attack database, according to the uniform resource locator of the data push request, unit information of the domain name and corresponding system website information are automatically filled in the target data.
6. The data access method according to claim 5, further comprising, if the target data is not historical attack data, the step of:
acquiring a unique identifier of the target data, an attack source IP and an attack target IP;
judging whether the target data is historical attack data of the attack database or not according to the unique identifier, the attack source IP and the attack target IP;
if the target data is historical attack data of the attack database, discarding the target data;
and if the target data is not the historical attack data of the attack database, judging that the target data is not the historical attack data.
7. The data access method according to claim 5, wherein after the unified mapping to the data dictionary corresponding to the category to which the unified mapping belongs, the method further comprises:
creating a corresponding database group for each engine in advance, wherein each database group is provided with a plurality of sub-libraries according to the data type;
matching a target database group according to the data source of the target data;
and matching corresponding target sub-libraries in the target database group according to the data types of the target data, and storing the mapped target data to the target sub-libraries.
8. A data access apparatus, comprising:
the data receiving module is used for placing the received target data into a corresponding target position according to the client information and the data type corresponding to the data pushing request when the data pushing request is received;
the data reading module is used for calling a corresponding target engine to read the target data from the target position according to the data source of the target data;
the data processing module is used for filling attack attribute information for the target data to obtain safe target data if the target data is not historical attack data;
and the standardization module is used for carrying out formatting cleaning on the safety target data according to the data source and uniformly mapping the safety target data to the data dictionary corresponding to the category to which the safety target data belongs.
9. An electronic device comprising a processor and a memory, the processor being configured to implement the steps of the data access method according to any one of claims 1 to 7 when executing a computer program stored in the memory.
10. A readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data access method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110909175.4A CN113626509A (en) | 2021-08-09 | 2021-08-09 | Data access method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110909175.4A CN113626509A (en) | 2021-08-09 | 2021-08-09 | Data access method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113626509A true CN113626509A (en) | 2021-11-09 |
Family
ID=78383686
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110909175.4A Withdrawn CN113626509A (en) | 2021-08-09 | 2021-08-09 | Data access method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113626509A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114201418A (en) * | 2021-12-13 | 2022-03-18 | 珠海格力电器股份有限公司 | Data access method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108881294A (en) * | 2018-07-23 | 2018-11-23 | 杭州安恒信息技术股份有限公司 | Attack source IP portrait generation method and device based on attack |
| CN109299174A (en) * | 2018-09-11 | 2019-02-01 | 北京奇安信科技有限公司 | A kind of multi-source information data aggregation processing method and device |
| CN111400061A (en) * | 2020-03-12 | 2020-07-10 | 泰康保险集团股份有限公司 | Data processing method and system |
| CN112019575A (en) * | 2020-10-22 | 2020-12-01 | 腾讯科技（深圳）有限公司 | Data packet processing method and device, computer equipment and storage medium |
| WO2021008028A1 (en) * | 2019-07-18 | 2021-01-21 | 平安科技（深圳）有限公司 | Network attack source tracing and protection method, electronic device and computer storage medium |
| CN112445854A (en) * | 2020-11-25 | 2021-03-05 | 平安普惠企业管理有限公司 | Multi-source business data real-time processing method and device, terminal and storage medium |
-
2021
- 2021-08-09 CN CN202110909175.4A patent/CN113626509A/en not_active Withdrawn
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108881294A (en) * | 2018-07-23 | 2018-11-23 | 杭州安恒信息技术股份有限公司 | Attack source IP portrait generation method and device based on attack |
| CN109299174A (en) * | 2018-09-11 | 2019-02-01 | 北京奇安信科技有限公司 | A kind of multi-source information data aggregation processing method and device |
| WO2021008028A1 (en) * | 2019-07-18 | 2021-01-21 | 平安科技（深圳）有限公司 | Network attack source tracing and protection method, electronic device and computer storage medium |
| CN111400061A (en) * | 2020-03-12 | 2020-07-10 | 泰康保险集团股份有限公司 | Data processing method and system |
| CN112019575A (en) * | 2020-10-22 | 2020-12-01 | 腾讯科技（深圳）有限公司 | Data packet processing method and device, computer equipment and storage medium |
| CN112445854A (en) * | 2020-11-25 | 2021-03-05 | 平安普惠企业管理有限公司 | Multi-source business data real-time processing method and device, terminal and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114201418A (en) * | 2021-12-13 | 2022-03-18 | 珠海格力电器股份有限公司 | Data access method and device, electronic equipment and storage medium |
| CN114201418B (en) * | 2021-12-13 | 2024-05-03 | 珠海格力电器股份有限公司 | Data access method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108734028B (en) | Data management method based on block chain, block chain link point and storage medium | |
| CN111414407A (en) | Data query method and device of database, computer equipment and storage medium | |
| KR102080156B1 (en) | Auto Recharge System, Method and Server | |
| CN105302920A (en) | Optimal management method and system for cloud storage data | |
| CN111556005A (en) | Authority management method, device, electronic equipment and storage medium | |
| CN109067732A (en) | Internet of things equipment and data insertion system, method and computer readable storage medium | |
| CN112446022A (en) | Data authority control method and device, electronic equipment and storage medium | |
| CN108399046B (en) | File operation request processing method and device | |
| CN113626509A (en) | Data access method and device, electronic equipment and readable storage medium | |
| CN110619204A (en) | Invitation code generation method and device, terminal equipment and storage medium | |
| CN115086047B (en) | Interface authentication method and device, electronic equipment and storage medium | |
| CN111831744A (en) | DAPP on-chain data retrieval system, method and medium | |
| CN110851853A (en) | Data isolation method and device, computer equipment and storage medium | |
| CN112416875B (en) | Log management method, device, computer equipment and storage medium | |
| CN114968822A (en) | Interface testing method and device, computer equipment and storage medium | |
| CN111698227A (en) | Information synchronization management method, device, computer system and readable storage medium | |
| WO2016039757A1 (en) | Information object system | |
| CN110413644B (en) | Data caching method, electronic device and computer readable storage medium | |
| CN114650436B (en) | Remote control method, device, equipment and medium based on background service | |
| CN115270110B (en) | Account inspection method and device, electronic equipment and storage medium | |
| CN115002211B (en) | Method, device, equipment and medium for realizing after-sale micro-service based on cloud protogenesis | |
| CN114614993B (en) | System interaction method and device, electronic equipment and storage medium | |
| CN109933573B (en) | Database service updating method, device and system | |
| CN115987497A (en) | HSS unauthorized access protection method, system, electronic equipment and storage medium | |
| CN116436784A (en) | Configuration file verification method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211109 |