CN113961595A - Data query method and device, computer equipment and storage medium - Google Patents
Data query method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113961595A CN113961595A CN202111216153.6A CN202111216153A CN113961595A CN 113961595 A CN113961595 A CN 113961595A CN 202111216153 A CN202111216153 A CN 202111216153A CN 113961595 A CN113961595 A CN 113961595A
- Authority
- CN
- China
- Prior art keywords
- data
- query result
- target
- monitored unit
- newly
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 89
- 238000012216 screening Methods 0.000 claims abstract description 84
- 238000012545 processing Methods 0.000 claims abstract description 76
- 238000001914 filtration Methods 0.000 claims abstract description 60
- 230000008569 process Effects 0.000 claims description 49
- 238000004590 computer program Methods 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 4
- 230000002776 aggregation Effects 0.000 description 3
- 238000004220 aggregation Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000000977 initiatory effect Effects 0.000 description 3
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data query method, a data query device, computer equipment and a storage medium. The method comprises the following steps: acquiring data streams of monitored units, and screening according to filtering and screening conditions in a preset data structure to obtain target data; counting the basic query results of each monitored unit based on the target data, and storing the basic query results into a preset data structure; and processing the basic query result in the preset data structure based on the data processing rule to obtain a final query result, and pushing the final query result to the terminal equipment so that the user can look up the final query result through the terminal equipment. By adopting the method, the data query efficiency can be improved.
Description
Technical Field
The present application relates to the field of database and data processing technologies, and in particular, to a data query method, an apparatus, a computer device, and a storage medium.
Background
With the explosion of network information technology, many different service systems are connected to the same data platform, and users query, monitor and process the service data of each service system (also called monitored unit) through the data platform.
Currently, a user performs query access on a data platform on the data of each monitored unit by sending a query request. Since the time of visiting each monitored unit is not uniform, it is generally necessary to set a query condition for each monitored unit. And in the query process, the data platform updates the user access time in the access time record in real time and only reserves the latest access time.
However, in the current data query method, different query screening conditions need to be set for each query of a monitored unit, so that the query screening conditions are complex, the query performance of the data platform is reduced, and since the data platform only keeps the latest access time of a user, the user needs to actively initiate a query request whenever the user needs to perform data query, and the data platform re-counts data query results according to the query request, so that the data query efficiency is low.
Disclosure of Invention
In view of the above, it is necessary to provide a data query method, apparatus, computer device and storage medium for solving the above technical problems.
A method of data query, the method comprising:
acquiring data streams of monitored units, and screening according to filtering and screening conditions in a preset data structure to obtain target data;
counting the basic query result of each monitored unit based on the target data, and storing the basic query result into the preset data structure;
and processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to terminal equipment so that a user can look up the final query result through the terminal equipment.
In one embodiment, the screening to obtain the target data according to the filtering and screening condition in the preset data structure includes:
periodically screening candidate data in each monitored unit data stream according to at least one filtering and screening condition in a preset data structure according to a preset time period, and classifying the candidate data according to at least one data classification rule to obtain classified target data;
the counting of the basic query result of each monitored unit based on the target data comprises:
aiming at the target data corresponding to each monitored unit, distributing a new data volume statistical process according to each event type and the number of the event types contained in the target data;
and counting the newly added data quantity of each event type in the target data through the newly added data quantity counting process to obtain a basic query result corresponding to each monitored unit.
In one embodiment, the basic query result includes a newly-added read data volume and a newly-added data volume, where the newly-added read data volume is a data volume that has been pushed to the terminal device in the newly-added data volume; processing the basic query result in the preset data structure based on the data processing rule to obtain a final query result, including:
acquiring a basic query result corresponding to each monitored unit in the preset data structure;
and calculating to obtain the newly-increased unread data volume of the monitored unit according to the newly-increased data volume and the newly-increased read data volume in the basic query result, and taking the newly-increased unread data volume as the final query result.
In one embodiment, the method further comprises:
receiving a data reading request, wherein the data reading request carries a target monitored unit identifier;
determining at least one filtering and screening condition and at least one data classification rule in the preset data structure according to the target monitored unit identifier;
the screening according to the filtering and screening conditions in the preset data structure to obtain the target data comprises the following steps:
screening candidate data from the data stream corresponding to the target monitored unit according to the filtering and screening conditions and the data classification rules;
classifying the candidate data according to the data classification rule to obtain classified target data; the target data comprises all data in a time range from the time of the last data reading request to the time of the current data reading request;
and reading the target data, and feeding the target data back to a display end of the terminal equipment for output and display.
In one embodiment, the counting the basic query result of each monitored unit based on the target data includes:
after the target data are fed back to the terminal equipment, marking the target data corresponding to the target monitored unit as read data to obtain newly-added read data;
and carrying out data volume statistics on the newly-added read data according to a data volume statistics process to obtain a basic query result of the target monitored unit.
In one embodiment, the performing data volume statistics on the newly-added read data according to a data volume statistics process to obtain a basic query result of the target monitored unit includes:
analyzing the event type of the network security event contained in the newly added read data;
and distributing a data volume statistical process according to the event types and the number of the event types, and determining the newly-increased read data volume of each event type through the data volume statistical process to obtain a basic query result of the target monitored unit.
A data query device, the device comprising:
the acquisition module is used for acquiring the data stream of each monitored unit and screening the data stream according to the filtering and screening conditions in the preset data structure to obtain target data;
the statistical module is used for counting the basic query result of each monitored unit based on the target data and storing the basic query result into the preset data structure;
and the pushing module is used for processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to the terminal equipment so as to enable a user to look up through the terminal equipment.
In one embodiment, the basic query result includes a read data volume and a newly-added data volume, where the newly-added read data volume is a data volume that has been pushed to the terminal device in the newly-added data volume; the pushing module is specifically configured to obtain, in the preset data structure, a basic query result corresponding to each monitored unit;
and calculating to obtain the newly-increased unread data volume of the monitored unit according to the newly-increased data volume and the newly-increased read data volume in the basic query result, and taking the newly-increased unread data volume as the final query result.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring data streams of monitored units, and screening according to filtering and screening conditions in a preset data structure to obtain target data;
counting the basic query result of each monitored unit based on the target data, and storing the basic query result into the preset data structure;
and processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to terminal equipment so that a user can look up the final query result through the terminal equipment.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring data streams of monitored units, and screening according to filtering and screening conditions in a preset data structure to obtain target data;
counting the basic query result of each monitored unit based on the target data, and storing the basic query result into the preset data structure;
and processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to terminal equipment so that a user can look up the final query result through the terminal equipment.
The data query method, the data query device, the computer equipment and the storage medium acquire data streams of monitored units, and target data are obtained through screening according to filtering and screening conditions in a preset data structure; counting the basic query result of each monitored unit based on the target data, and storing the basic query result into the preset data structure; and processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to terminal equipment so that a user can look up the final query result through the terminal equipment. By adopting the method, the data in the data stream is subjected to real-time statistics of basic query results through the filtering and screening conditions in the preset data structure, and then the final query results of each monitored unit are counted according to the preset data processing rule and the basic query results, and are pushed to the terminal equipment of the user. The final query result of each monitored unit in the push data stream is updated in real time without initiating a data query request by a user or recalculating the query result for each data query request, so that the data query efficiency is improved.
Drawings
FIG. 1 is a flow diagram illustrating a data query method according to one embodiment;
FIG. 2 is a schematic flow chart diagram illustrating steps for monitoring and updating basic query results in real-time in one embodiment;
FIG. 3 is a flowchart of the new data filtering, screening and statistics steps in one embodiment;
FIG. 4 is a flowchart illustrating specific steps in determining a final query result in one embodiment;
FIG. 5 is a flow chart illustrating the data reading step in one embodiment;
FIG. 6 is a flow chart illustrating the data statistics performed by the target monitoring unit in one embodiment;
FIG. 7 is a flowchart of the steps performed in one embodiment to mark read data and compute a base query result;
FIG. 8 is a flowchart of the step of determining a base query result for a target monitored unit, under one embodiment;
FIG. 9 is a flow diagram of an example of a method for querying data in one embodiment;
FIG. 10 is a block diagram of a data query method in one embodiment;
FIG. 11 is a block diagram showing the structure of a data search device according to an embodiment;
FIG. 12 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, a data query method is provided, which may be applied to a terminal, a server, or a system including the terminal and the server, and is implemented by interaction between the terminal and the server. In this embodiment, the method is applied to a terminal for example, wherein a data query processing platform is integrated on the terminal, and therefore, in each embodiment, the terminal executes each step through the data query processing platform. In this embodiment, the method includes the steps of:
In implementation, data (e.g., network security event data) of each monitored unit is uniformly accessed to the data query processing platform, and the data query processing platform performs planning processing, so that the data query processing platform may obtain a data stream of each monitored unit and filter data in the data stream according to a filtering and screening condition in a preset data structure, for example, the filtering and screening condition in the preset data structure may include an event level, an event type, time information, and the like of a network security event of the monitored unit, and the filtering and screening condition of the data is not limited in the embodiment of the present application. And then the data query processing platform screens the target data according to the filtering and screening conditions in the preset data structure.
And 102, counting the basic query results of each monitored unit based on the target data, and storing the basic query results into a preset data structure.
In implementation, the data query processing platform counts the basic query result of each monitored unit based on the target data obtained by screening. Wherein the base query result may also be referred to as an intermediate calculation result of the data query. The determination of the intermediate calculation result depends on the intermediate calculation output of the final calculation result, for example, if the final calculation result is a new unread data volume, the intermediate calculation result of the parameter includes the total number of new data and the read data volume of each monitored unit. And if the final calculation result is the total quantity of the network security event data, the intermediate calculation result of the parameter contains the total quantity of the network security event data of each monitored unit. Therefore, the present application does not limit the basic query result.
And 103, processing the basic query result in the preset data structure based on the data processing rule to obtain a final query result, and pushing the final query result to the terminal equipment so that the user can look up the final query result through the terminal equipment.
In implementation, in a preset time period, the data query processing platform performs statistics to obtain a final query result of each monitored unit according to a preset data processing rule and a basic query result of each monitored unit. Specifically, the data query processing platform updates the basic query result of each monitored unit in real time (for example, 5 seconds is one period), and further updates the final query result according to the updated basic query result at preset time intervals (15 seconds is one period). The data processing rule represents a user query requirement, for example, if the user query requirement is a newly-added unread data volume, the data query processing platform may process the basic query result according to the data processing rule, and obtain a final query result as the newly-added unread data volume. Then, the data query processing platform stores the generated final query result in a Remote Dictionary service (Redis) database, and pushes the final query result to a web front end of the terminal device for a preset time period (for example, 15 seconds) so as to enable the user to perform real-time query. Specifically, the data query processing platform monitors the final query result in the Redis database in real time through the websocket, and pushes the monitored final query result to the web front end in real time.
In the data query method, a data query processing platform acquires data streams of monitored units, and target data are obtained by screening according to filtering and screening conditions in a preset data structure; counting the basic query results of each monitored unit based on the target data, and storing the basic query results into a preset data structure; and processing the basic query result in the preset data structure based on the data processing rule to obtain a final query result, and pushing the final query result to the terminal equipment so that the user can look up the final query result through the terminal equipment. By adopting the method, the data in the data stream is subjected to real-time statistics of basic query results through the filtering and screening conditions in the preset data structure, and then the final query results of each monitored unit are counted according to the preset data processing rule and the basic query results, and are pushed to the terminal equipment of the user. The final query result of each monitored unit in the push data stream is updated in real time without initiating a data query request by a user or recalculating the query result for each data query request, so that the data query efficiency is improved.
In one embodiment, as shown in fig. 2, the step 101 of filtering the target data according to the filtering condition in the preset data structure includes:
In implementation, the data query processing platform monitors a data structure in the Redis database in real time, and performs real-time screening on data streams of monitored units according to filtering and screening conditions contained in the data structure based on a preset time period to obtain candidate data. And classifying the candidate data according to the data classification rule to obtain target data. Specifically, the data query processing platform periodically filters in each monitored unit data stream based on at least one filtering and screening condition in a data structure of the database Redis according to a preset time period (for example, 15 seconds), so as to obtain screened candidate data, and further classifies the candidate data according to at least one data classification rule (for example, an event level data classification rule), so as to obtain classified target data. As shown in fig. 3, the filtering conditions included in the data structure are: time screening conditions (day: 0 to 24), monitored unit screening conditions (effective monitored units: monitored unit a, monitored unit B, and monitored unit C), and the like. The data classification rule comprises: a first classification rule: the effective monitored unit, the second classification rule: the event types and the like, therefore, the data query processing platform can not only filter and screen the data in the data stream, but also perform aggregate classification on the screened data, and the embodiment of the application does not limit the types and the number of the data screening conditions and the data classification rules included in the data structure.
Optionally, in the process of data screening and processing performed by the data query processing platform, the filtering and screening condition may be selected in advance to match the data classification rule, and the matched filtering and screening condition and the matched data classification rule are encapsulated into one component.
The specific processing procedure of step 102 includes:
In implementation, the data query processing platform allocates, for target data corresponding to each monitored unit, a new data volume statistical process according to each event type and event type data included in the target data, for example, the target data includes 5 event types, and the data volume corresponding to each event type is: event type 1: 100 strips; event type 2: 80 strips; event type 3: 50 strips; event type 4: 20 strips; event type 5: 10 strips. Based on the 5 event types, 3 data volume statistical processes can be appropriately allocated to perform statistics on data, and according to the data volume of each event type, load balance among the data volume statistical processes is ensured, for example, process 1 counts the data volume of event type 1, process 2 counts the data volume of event type 2, and process 3 counts the data volumes of event type 3, event type 4, and event type 5.
As shown in fig. 3, N (N ═ 10) new data volume statistics processes are allocated to 10 event types, and data volume statistics is performed on data of each event type. N may be equal to the number of the event types, or may be greater than or less than the number of the event types, which is not limited in the embodiments of the present application.
And step 203, counting the newly added data volume of each event type in the target data through the newly added data volume counting process to obtain a basic query result corresponding to each monitored unit.
In implementation, the data query processing platform counts the new data volume of each event type in the target data through a new data volume counting process, for example, event type a: 50, event type B: 150, event type C: 40, event type D: and 120, taking the result as a basic query result (newly increased data volume query result) corresponding to each monitored unit.
In one embodiment, as shown in fig. 4, the basic query result includes a newly added read data volume and a newly added data volume, where the newly added read data volume is a data volume that has been pushed to the terminal device in the newly added data volume; the specific processing procedure of step 103 includes the following steps:
In implementation, the data query processing platform plans a data structure for storing data in advance, and stores the planned data structure in the memory database Redis, where the data structure includes not only data in a data stream, but also a filtering and screening condition (e.g., a filtering and screening attribute value: event level) of the data and a basic query result, specifically, the data is screened and stored based on the filtering and screening condition, and the basic query result is obtained by calculation based on the screened target data. The basic query result may also be referred to as an intermediate calculation result, for example, a total data volume of the newly added network security events, a data volume of the newly added read network security events, and the like, and then the data query processing platform monitors the data structure in the Redis database through the websocket, and obtains the basic query result in the target data structure corresponding to each monitored unit in a preset time period.
Optionally, the data query processing platform may push the basic query result to a web end of the terminal device for output and display according to a preset time period.
In the implementation, the data query processing platform performs difference calculation on the newly added data volume (newly added total data volume) and the newly added read data volume according to the newly added data volume and the newly added read data volume in the basic query result to obtain the newly added unread data volume of the monitored unit, and the newly added unread data volume is used as the final query result of the monitored unit.
In this embodiment, the data query processing platform monitors the data structure in the Redis database in real time, and performs real-time statistics on the final query result of each monitored unit by using the basic query result of each monitored unit included in the data structure, so as to push the final query result to the user for output and display, without initiating a data query request by the user, thereby improving the data query efficiency.
In one embodiment, as shown in fig. 5, when a user needs to read data in a data stream of a monitored unit, a data reading request needs to be initiated, the method further includes:
And the data reading request carries the target monitored unit identification.
In implementation, when a user needs to read data of a certain monitored unit, the user sends a data reading request to a background server through a terminal device web end. And then, the data query processing platform receives the data reading request and identifies the monitored unit to be read according to the data reading request.
In implementation, the data query processing platform identifies a target data structure in the overall data structure according to the target monitored unit identifier, and determines at least one target filtering and screening condition of the target monitored unit in the target data structure, for example, as shown in fig. 6, the determined target filtering and screening condition is a time condition: starting time (0) to ending time (24), the monitored unit (for example, monitored unit a) is valid, and then, the data classification rule (also referred to as aggregation rule) group by () matched with the target filtering and screening condition (for example, network security event level classification rule) is determined according to the target filtering condition. The data classification rule (aggregation rule) is a rule for performing classification statistics on data, for example, the data classification rule is a network security event level, and if the network security event level is divided into 3 levels, the data of the same monitored unit is divided by the first level of the network security event, the second level of the network security event and the third level of the network security event.
The specific processing procedure of step 101 includes:
and 503, screening the data stream corresponding to the target monitored unit according to the filtering and screening conditions and the data classification rules to obtain candidate data.
In implementation, the data query processing platform screens candidate data from the data stream corresponding to the target monitored unit according to the filtering and screening conditions and the data classification rules. For example, if the filtering condition includes effective monitoring unit information (i.e., target monitored unit) and a data time filtering condition defined by the time of day (0 hour to 24 hours), all the new data up to the time of sending the data reading request (e.g., 8 days) (the subsequent time period has not yet come) are filtered and obtained as candidate data.
And step 504, classifying the candidate data according to the data classification rule to obtain classified target data.
The target data comprises all data in a time range from the time of the last data reading request to the time of the current data reading request.
In implementation, the data query processing platform classifies the candidate data according to the data classification rule matched with the filtering and screening condition to obtain the classified target data. For example, if the data classification rule is an event level classification rule, the candidate data may be classified according to the event level classification rule, so as to obtain target data classified by a security event level in the target monitored unit data stream.
And 505, reading the target data, and feeding the target data back to the display end of the terminal equipment for output and display.
In implementation, the data query processing platform reads the target data, and feeds the target data back to the terminal device display terminal for output and display, so that a user can read and further process the data of the target monitored unit.
In one embodiment, as shown in fig. 7, the specific process of step 102 includes the following steps:
In implementation, after the target data is fed back to the terminal device, the data query processing platform marks the read target data of the target monitored unit as read, and new read data is obtained. Specifically, for newly added read data of a certain monitored unit (target monitored unit), the newly added unread data amount recorded in the data structure is updated to zero. For example, when the new data volume on the day starts to be counted from 0, and the user sends a data reading request at 8 am, the data query result of the target monitored unit is pushed to the web end in real time (15 seconds) in the time range from 0 to 8 hours, and when the data query result reaches 8, the data query result is updated as: 500 new data volumes are added, 500 new unread data volumes are added, and 0 new read data volume is added. Meanwhile, after a user initiates a data reading request at 8, all unread data are read and pushed in response to the data reading request, so that all unread data are marked as read, and the data query result is updated as follows: the new data volume is 500, the new unread data volume is 0, and the new read data volume is 500, namely the obtained read data volume is 500.
And step 702, carrying out data volume statistics on the newly added read data according to the data volume statistics process to obtain a basic query result of the target monitored unit.
In implementation, the data query processing platform allocates a corresponding data volume statistical process to the acquired target data, and performs data volume statistics on the target data (newly added read data) according to the data volume statistical process to obtain a basic query result (i.e., newly added read data) of the target monitored unit.
In one embodiment, as shown in fig. 8, the specific processing steps of step 702 include:
In implementation, the data query processing platform may further perform division processing on the obtained newly-added read data to meet various data query requirements of the user, as shown in fig. 6, when the newly-added read data is network security data, the network security data includes multiple different network security event types (for example, 10 types), and further, the data query processing platform may analyze the network security event types included in the newly-added read data to further divide the newly-added read data according to the network security event types.
In implementation, the data query processing platform performs distribution of data volume statistics processes according to each event type and the data volume of the event type, for example, newly added read data includes 10 event types, 3 data volume statistics processes can be appropriately distributed to perform statistics on the data based on the 10 event types, and load balance among the data volume statistics processes is ensured according to the data volume of each event type. And then, the data query processing platform obtains the newly-increased read data volume of each event type according to the statistics of the data volume statistical process, and the newly-increased read data volume is used as a basic query result of the target monitored unit.
Optionally, the data processing platform updates the recalculated basic query result to a read field of a data structure in which the target monitored unit is located in the Redis database, so as to update and push the basic query result to the user web end for output and display in real time.
It should be understood that, although the steps in the flowcharts of fig. 1 to 8 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1 to 8 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the other steps or stages.
In an embodiment, as shown in fig. 9, an example of a data query method is provided, where a specific implementation architecture of the example is shown in fig. 10, and the specific implementation architectures shown in fig. 9 and fig. 10 include the following steps:
And 904, pushing the updated target data, the basic query result and the final query result to the web front end in real time in a websocket mode.
In one embodiment, as shown in fig. 11, there is provided a data query apparatus 1100, including: an obtaining module 1110, a counting module 1120, and a pushing module 1130, wherein:
an obtaining module 1110, configured to obtain data streams of each monitored unit, and obtain target data by screening according to filtering and screening conditions in a preset data structure;
a statistics module 1120, configured to count basic query results of each monitored unit based on the target data, and store the basic query results in the preset data structure;
the pushing module 1130 is configured to process the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and push the final query result to a terminal device, so that a user can look up the final query result through the terminal device.
In one embodiment, the obtaining module 1110 is specifically configured to periodically filter candidate data in each monitored unit data stream according to at least one filtering and screening condition in a preset data structure according to a preset time period, and classify the candidate data according to at least one data classification rule to obtain classified target data;
a statistical module 1120, configured to allocate, for the target data corresponding to each monitored unit, a new data volume statistical process according to each event type and the number of event types included in the target data; and counting the newly added data quantity of each event type in the target data through the newly added data quantity counting process to obtain a basic query result corresponding to each monitored unit.
In one embodiment, the basic query result includes a newly-added read data volume and a newly-added data volume, where the newly-added read data volume is a data volume that has been pushed to the terminal device in the newly-added data volume; the pushing module 1130 is specifically configured to obtain, in the preset data structure, a basic query result corresponding to each monitored unit;
and calculating to obtain the newly-increased unread data volume of the monitored unit according to the newly-increased data volume and the newly-increased read data volume in the basic query result, and taking the newly-increased unread data volume as the final query result.
In one embodiment, the apparatus 1100 further comprises:
the receiving module is used for receiving a data reading request, and the data reading request carries the target monitored unit identifier;
the determining module is used for determining at least one filtering and screening condition and at least one data classification rule in the preset data structure according to the target monitored unit identifier;
the statistical module 1120 is further configured to screen a data stream corresponding to the target monitored unit according to the filtering and screening condition to obtain candidate data;
classifying the candidate data according to the data classification rule to obtain classified target data; the target data comprises all data in a time range from the time of the last data reading request to the time of the current data reading request;
and reading the target data, and feeding the target data back to a display end of the terminal equipment for output and display.
In one embodiment, the statistical module 1120 is further configured to mark the target data corresponding to the target monitored unit as read after the target data is fed back to the terminal device, so as to obtain new read data;
and carrying out data volume statistics on the newly-added read data according to a data volume statistics process to obtain a basic query result of the target monitored unit.
In one embodiment, the statistical module 1120 is further configured to parse an event type of a network security event included in the newly added read data;
and distributing a data volume statistical process according to the event types and the number of the event types, and determining the newly-increased read data volume of each event type through the data volume statistical process to obtain a basic query result of the target monitored unit.
For specific limitations of the data query device 1100, reference may be made to the above limitations of the data query method, which are not described herein again. The various modules in the data query apparatus 1100 described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 12. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data query method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 12 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
acquiring data streams of monitored units, and screening according to filtering and screening conditions in a preset data structure to obtain target data;
counting the basic query result of each monitored unit based on the target data, and storing the basic query result into the preset data structure;
and processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to terminal equipment so that a user can look up the final query result through the terminal equipment.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
periodically screening candidate data in each monitored unit data stream according to at least one filtering and screening condition in a preset data structure according to a preset time period, and classifying the candidate data according to at least one data classification rule to obtain classified target data;
aiming at the target data corresponding to each monitored unit, distributing a new data volume statistical process according to each event type and the number of the event types contained in the target data;
and counting the newly added data quantity of each event type in the target data through the newly added data quantity counting process to obtain a basic query result corresponding to each monitored unit.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring a basic query result corresponding to each monitored unit in the preset data structure;
and calculating to obtain the newly-increased unread data volume of the monitored unit according to the newly-increased data volume and the newly-increased read data volume in the basic query result, and taking the newly-increased unread data volume as the final query result.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
receiving a data reading request, wherein the data reading request carries a target monitored unit identifier;
determining at least one filtering and screening condition and at least one data classification rule in the preset data structure according to the target monitored unit identifier;
screening candidate data from the data stream corresponding to the target monitored unit according to the filtering and screening conditions;
classifying the candidate data according to the data classification rule to obtain classified target data; the target data comprises all data in a time range from the time of the last data reading request to the time of the current data reading request;
and reading the target data, and feeding the target data back to a display end of the terminal equipment for output and display.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
after the target data are fed back to the terminal equipment, marking the target data corresponding to the target monitored unit as read data to obtain newly-added read data;
and carrying out data volume statistics on the newly-added read data according to a data volume statistics process to obtain a basic query result of the target monitored unit.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
analyzing the event type of the network security event contained in the newly added read data;
and distributing a data volume statistical process according to the event types and the number of the event types, and determining the newly-increased read data volume of each event type through the data volume statistical process to obtain a basic query result of the target monitored unit.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for data query, the method comprising:
acquiring data streams of monitored units, and screening according to filtering and screening conditions in a preset data structure to obtain target data;
counting the basic query result of each monitored unit based on the target data, and storing the basic query result into the preset data structure;
and processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to terminal equipment so that a user can look up the final query result through the terminal equipment.
2. The method according to claim 1, wherein the screening target data according to the filtering and screening conditions in the preset data structure comprises:
periodically screening candidate data in each monitored unit data stream according to at least one filtering and screening condition in a preset data structure according to a preset time period, and classifying the candidate data according to at least one data classification rule to obtain classified target data;
the counting of the basic query result of each monitored unit based on the target data comprises:
aiming at the target data corresponding to each monitored unit, distributing a new data volume statistical process according to each event type and the number of the event types contained in the target data;
and counting the newly added data quantity of each event type in the target data through the newly added data quantity counting process to obtain a basic query result corresponding to each monitored unit.
3. The method according to claim 1, wherein the basic query result includes a newly added read data volume and a newly added data volume, and the newly added read data volume is a data volume that has been pushed to the terminal device in the newly added data volume; processing the basic query result in the preset data structure based on the data processing rule to obtain a final query result, including:
acquiring a basic query result corresponding to each monitored unit in the preset data structure;
and calculating to obtain the newly-increased unread data volume of the monitored unit according to the newly-increased data volume and the newly-increased read data volume in the basic query result, and taking the newly-increased unread data volume as the final query result.
4. The method of claim 1, further comprising:
receiving a data reading request, wherein the data reading request carries a target monitored unit identifier;
determining at least one filtering and screening condition and at least one data classification rule in the preset data structure according to the target monitored unit identifier;
the screening according to the filtering and screening conditions in the preset data structure to obtain the target data comprises the following steps:
screening candidate data from the data stream corresponding to the target monitored unit according to the filtering and screening conditions;
classifying the candidate data according to the data classification rule to obtain classified target data; the target data comprises all data in a time range from the time of the last data reading request to the time of the current data reading request;
and reading the target data, and feeding the target data back to a display end of the terminal equipment for output and display.
5. The method of claim 4, wherein said counting the base query results for each of the monitored units based on the target data comprises:
after the target data are fed back to the terminal equipment, marking the target data corresponding to the target monitored unit as read data to obtain newly-added read data;
and carrying out data volume statistics on the newly-added read data according to a data volume statistics process to obtain a basic query result of the target monitored unit.
6. The method of claim 5, wherein the performing data volume statistics on the newly-added read data according to a data volume statistics process to obtain a basic query result of the target monitored unit comprises:
analyzing the event type of the network security event contained in the newly added read data;
and distributing a data volume statistical process according to the event types and the number of the event types, and determining the newly-increased read data volume of each event type through the data volume statistical process to obtain a basic query result of the target monitored unit.
7. A data query apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring the data stream of each monitored unit and screening the data stream according to the filtering and screening conditions in the preset data structure to obtain target data;
the statistical module is used for counting the basic query result of each monitored unit based on the target data and storing the basic query result into the preset data structure;
and the pushing module is used for processing the basic query result in the preset data structure based on a data processing rule to obtain a final query result, and pushing the final query result to the terminal equipment so as to enable a user to look up through the terminal equipment.
8. The apparatus according to claim 7, wherein the basic query result includes a read data volume and a new data volume, and the new read data volume is a data volume that has been pushed to the terminal device in the new data volume; the pushing module is specifically configured to obtain, in the preset data structure, a basic query result corresponding to each monitored unit;
and calculating to obtain the newly-increased unread data volume of the monitored unit according to the newly-increased data volume and the newly-increased read data volume in the basic query result, and taking the newly-increased unread data volume as the final query result.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216153.6A CN113961595A (en) | 2021-10-19 | 2021-10-19 | Data query method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216153.6A CN113961595A (en) | 2021-10-19 | 2021-10-19 | Data query method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113961595A true CN113961595A (en) | 2022-01-21 |
Family
ID=79465383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111216153.6A Pending CN113961595A (en) | 2021-10-19 | 2021-10-19 | Data query method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113961595A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103250147A (en) * | 2010-10-14 | 2013-08-14 | 惠普发展公司，有限责任合伙企业 | Continuous querying of a data stream |
| CN105991692A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Information pushing method, information display method, and related apparatus and system |
| CN107391746A (en) * | 2017-08-10 | 2017-11-24 | 深圳前海微众银行股份有限公司 | Log analysis method, equipment and computer-readable recording medium |
| CN112084217A (en) * | 2020-09-17 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
| CN112100219A (en) * | 2020-09-22 | 2020-12-18 | 平安养老保险股份有限公司 | Report generation method, device, equipment and medium based on database query processing |
| CN112133388A (en) * | 2019-06-25 | 2020-12-25 | Iqvia 有限公司 | Computer-implemented method and system |
| CN112860709A (en) * | 2021-03-17 | 2021-05-28 | 网易（杭州）网络有限公司 | Service index query method, device, equipment and storage medium |
-
2021
- 2021-10-19 CN CN202111216153.6A patent/CN113961595A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103250147A (en) * | 2010-10-14 | 2013-08-14 | 惠普发展公司，有限责任合伙企业 | Continuous querying of a data stream |
| CN105991692A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Information pushing method, information display method, and related apparatus and system |
| CN107391746A (en) * | 2017-08-10 | 2017-11-24 | 深圳前海微众银行股份有限公司 | Log analysis method, equipment and computer-readable recording medium |
| CN112133388A (en) * | 2019-06-25 | 2020-12-25 | Iqvia 有限公司 | Computer-implemented method and system |
| CN112084217A (en) * | 2020-09-17 | 2020-12-15 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
| CN112100219A (en) * | 2020-09-22 | 2020-12-18 | 平安养老保险股份有限公司 | Report generation method, device, equipment and medium based on database query processing |
| CN112860709A (en) * | 2021-03-17 | 2021-05-28 | 网易（杭州）网络有限公司 | Service index query method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107391538B (en) | Click data acquisition, processing and display method, device, equipment and storage medium | |
| CN111782966B (en) | User grouping method, device, computer equipment and medium | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| CN110399380B (en) | Data processing method, electronic device and storage medium | |
| CN107656807B (en) | Automatic elastic expansion method and device for virtual resources | |
| US20170161306A1 (en) | Method and apparatus for data quality management and control | |
| CN113342603B (en) | Alarm data processing method and device, computer equipment and storage medium | |
| CN107807967B (en) | Real-time recommendation method, electronic device and computer-readable storage medium | |
| CN110704675B (en) | Object management method, device, computer equipment and storage medium | |
| US20140006324A1 (en) | Automatic event analysis | |
| CN111382334B (en) | Data processing method and device, computer and readable storage medium | |
| CN110580293A (en) | Entity relationship storage method and device | |
| CN110737673B (en) | Data processing method and system | |
| CN110543509B (en) | Monitoring system, method and device for user access data and electronic equipment | |
| CN111737233A (en) | Data monitoring method and device | |
| CN109542947B (en) | Data statistical method, device, computer equipment and storage medium | |
| CN113961595A (en) | Data query method and device, computer equipment and storage medium | |
| CN110932935A (en) | Resource control method, device, equipment and computer storage medium | |
| CN108399031B (en) | Method and device for determining interface layout mode | |
| US20160147838A1 (en) | Receiving node, data management system, data management method and strage medium | |
| CN113835953A (en) | Statistical method and device of job information, computer equipment and storage medium | |
| CN113836130A (en) | Data quality evaluation method, device, equipment and storage medium | |
| CN111131393B (en) | User activity data statistical method, electronic device and storage medium | |
| CN110941608A (en) | Method, device and equipment for generating buried point analysis and funnel analysis report | |
| CN110837951A (en) | Business channel sequencing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: Room 332, 3 / F, Building 102, 28 xinjiekouwei street, Xicheng District, Beijing 100088 Applicant after: QAX Technology Group Inc. Applicant after: Qianxin Wangshen information technology (Beijing) Co.,Ltd. Address before: Room 332, 3 / F, Building 102, 28 xinjiekouwei street, Xicheng District, Beijing 100088 Applicant before: QAX Technology Group Inc. Country or region before: China Applicant before: LEGENDSEC INFORMATION TECHNOLOGY (BEIJING) Inc. |