CN116866213A - Flow distribution acquisition method, device, equipment and storage medium - Google Patents

Flow distribution acquisition method, device, equipment and storage medium Download PDF

Info

Publication number
CN116866213A
CN116866213A CN202310897129.6A CN202310897129A CN116866213A CN 116866213 A CN116866213 A CN 116866213A CN 202310897129 A CN202310897129 A CN 202310897129A CN 116866213 A CN116866213 A CN 116866213A
Authority
CN
China
Prior art keywords
target
flow distribution
values
preset
occurrence
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310897129.6A
Other languages
Chinese (zh)
Inventor
杨昭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Mengju Information Technology Co ltd
Original Assignee
Shanghai Weimeng Enterprise Development Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Weimeng Enterprise Development Co ltd filed Critical Shanghai Weimeng Enterprise Development Co ltd
Priority to CN202310897129.6A priority Critical patent/CN116866213A/en
Publication of CN116866213A publication Critical patent/CN116866213A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/06—Generation of reports
    • H04L43/062—Generation of reports related to network traffic

Abstract

The application discloses a flow distribution acquisition method, a device, equipment and a storage medium, which relate to the technical field of computers and comprise the following steps: collecting flow information of an interface to be counted, storing the flow information into a flow distribution key value pair set, and determining whether the current flow distribution key value pair set is in a statistically available state or not; if yes, counting the occurrence times of the values of the target fields and updating the latest occurrence time; determining the sequence of the values of all target fields based on a preset rule, determining the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks based on the sequence, and deleting the values of the target fields after the target ranks; the method comprises the steps of obtaining a value of a target field and the occurrence number of the value of the target field, which meet preset conditions, and storing the value of the target field and the occurrence number of the value of the target field, which meet preset conditions, into a database. Therefore, the application can acquire the flow distribution of the interface on the premise of using as few resources as possible.

Description

Flow distribution acquisition method, device, equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a storage medium for acquiring flow distribution.
Background
The prior art scene is divided into two parts as shown in fig. 1, and the results of dotting reporting and analysis are: 1. logging operation and maintenance writes the log to kafka through filecoat in the container through each called entry of the log record interface, and log center consumes kafka to be written into an elastic search (real-time search engine); 2. and searching data in the elastic search by kibana, and aggregating to obtain the occurrence number and the ranking of the value of the attention field of a certain interval. However, the above technology has some defects, only the original data is unprocessed, and when the statistics is performed, the queries are required to be aggregated under a large amount of data, so that the performance is low; generating a large amount of logs, and needing to pay a large storage cost; to reduce storage costs, data that is relatively long will be lost. Therefore, how to obtain the traffic distribution of the interface on the premise of using as few resources as possible is a problem to be solved in the art.
Disclosure of Invention
In view of the above, an object of the present application is to provide a traffic distribution acquiring method, apparatus, device, and storage medium, which can acquire a traffic distribution of an interface, that is, a rank of frequency of occurrence of different values of a field of interest in a parameter, on the premise of using as little resources as possible. The specific scheme is as follows:
in a first aspect, the present application discloses a flow distribution acquisition method, including:
collecting flow information of an interface to be counted based on a tangent plane programming technology, storing the flow information into a flow distribution key value pair set, judging the content in the flow distribution key value pair set, and determining whether the current flow distribution key value pair set is in a statistically available state or not;
if yes, counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time;
determining the sequence of the values of the target fields in the preset time period based on a preset rule, determining the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking based on the sequence, and deleting the values of the target fields after the target ranking according to the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking;
and acquiring the value of the target field and the occurrence frequency of the value of the target field, which are in accordance with a preset condition, in the flow distribution key value pair set within a preset statistical time, and storing the value of the target field and the occurrence frequency of the value of the target field, which are in accordance with the preset condition, into a database.
Optionally, the collecting flow information of the interface to be counted based on the tangent plane programming technology includes:
and adding a predefined annotation on the interface to be counted based on a tangent plane programming technology so as to acquire the flow information of the interface to be counted.
Optionally, the determining whether the current flow distribution key value pair set is in a statistically available state includes:
if the current flow distribution key value pair set does not have flow distribution data in a preset historical time period, judging that the current flow distribution key value pair set is in a statistical state;
if the number of the different values corresponding to the target field in the current flow distribution key value pair set does not exceed the preset threshold, judging that the current flow distribution key value pair set is in a statistical state.
Optionally, the determining whether the current flow distribution key value pair set is in a statistically available state further includes:
if the current flow distribution key value pair set has flow distribution data in a preset historical time period or the number of different values corresponding to the target fields in the current flow distribution key value pair set exceeds a preset threshold, corresponding fusing processing is performed to prohibit statistics on the values of all the target fields in the current flow distribution key value pair set.
Optionally, the determining the ordering of the values of the target fields in the preset time period based on the preset rule includes:
and sorting the values of the target fields according to the preset timing screening time based on the occurrence times of the values of the target fields in the preset time period, and if the occurrence times of the values of the target fields are the same, sorting the values of the target fields according to the latest occurrence time corresponding to the values of the target fields so as to determine the sorting of the values of the target fields.
Optionally, before deleting the value of the target field after the target ranking according to the occurrence number of the value of the target field corresponding to the target ranking and the latest occurrence time, the method further includes:
comparing the occurrence times corresponding to the values of the target fields after the target ranking with the latest occurrence time, and obtaining a comparison result;
deleting the value of the target field which accords with a preset deleting condition based on the comparison result, and not processing the value of the target field which does not accord with the preset deleting condition; the preset deleting condition is that the occurrence frequency corresponding to the value of the target field after the target ranking is smaller than the occurrence frequency of the value of the target field corresponding to the target ranking.
Optionally, the storing the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, in a database includes:
and assembling the value of the target field and the occurrence frequency of the value of the target field which meet the preset condition into an insert statement, and storing the value of the target field and the occurrence frequency of the value of the target field which meet the preset condition into a relational database management system based on the insert statement.
In a second aspect, the present application discloses a flow distribution acquisition device, including:
the judging module is used for collecting flow information of an interface to be counted based on a tangent plane programming technology, storing the flow information into a flow distribution key value pair set, judging the content in the flow distribution key value pair set and determining whether the current flow distribution key value pair set is in a statistically available state or not;
the statistics and updating module is used for counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time if the flow distribution key value pair values of all target fields are in the same time period;
a deleting module, configured to determine an order of values of the target fields in the preset time period based on a preset rule, determine the occurrence number and the latest occurrence time of the values of the target fields corresponding to a target ranking based on the order, and delete the values of the target fields after the target ranking according to the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking;
the storage module is used for acquiring the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, in the flow distribution key value pair set within the preset statistical time, and storing the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, into a database.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
and a processor for executing the computer program to implement the steps of the flow distribution acquisition method disclosed above.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the steps of the flow distribution acquisition method disclosed above.
When the flow distribution is acquired, firstly, collecting flow information of an interface to be counted based on a tangent plane programming technology, storing the flow information into a flow distribution key value pair set, judging the content in the flow distribution key value pair set, and determining whether the current flow distribution key value pair set is in a statistically available state or not; if yes, counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time; then determining the sequence of the values of the target fields in the preset time period based on a preset rule, determining the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks based on the sequence, and deleting the values of the target fields after the target ranks according to the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks; and finally, acquiring the value of the target field and the occurrence number of the value of the target field, which are in accordance with a preset condition, in the flow distribution key value pair set within a preset statistical time, and storing the value of the target field and the occurrence number of the value of the target field, which are in accordance with the preset condition, into a database. The application can be seen that the flow information is stored into the flow distribution key value pair set, when the flow distribution key value pair set is in a statistical state, the occurrence times of the values of all the target fields are counted and the latest occurrence time is updated, the values of all the target fields are sequenced after the counting and updating, so that the values of the target fields corresponding to the target ranks are determined, the values of the target fields after the target ranks can be deleted, finally, the values of all the target fields after the deletion are selected again, and the selected values of the target fields meeting the preset conditions and the occurrence times of the values of the target fields are stored into the database. Therefore, the application can compress the stored data to the minute dimension, only keep the head flow data needing to be concerned to the database, and facilitate the follow-up visual inquiry according to the time range and the multidimensional field of interest. And meanwhile, the storage cost is greatly reduced. As the stored information is reduced, the historical data does not need to be deleted, and the historical data can be checked at any time.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a flow distribution acquisition method in the prior art;
FIG. 2 is a flow chart of a flow distribution acquisition method disclosed by the application;
FIG. 3 is a flow chart of a specific flow distribution acquisition method disclosed in the present application;
fig. 4 is a schematic structural diagram of a flow distribution acquiring device according to the present application;
fig. 5 is a block diagram of an electronic device according to the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The prior art scene is divided into two parts, namely, dotting reporting and analyzing results: 1. logging operation and maintenance writes the log to kafka through filecoat in the container through each called entry of the log record interface, and log center consumes kafka to be written into an elastic search (real-time search engine); 2. and searching data in the elastic search by kibana, and aggregating to obtain the occurrence number and the ranking of the value of the attention field of a certain interval. However, the above technology has some defects, only the original data is unprocessed, and when the statistics is performed, the queries are required to be aggregated under a large amount of data, so that the performance is low; generating a large amount of logs, and needing to pay a large storage cost; to reduce storage costs, data that is relatively long will be lost. In order to solve the problems, the application can acquire the traffic distribution of the interface, namely the frequency ranking of the occurrence of different values of the concerned field in the participation under the premise of using as little resources as possible.
Referring to fig. 2, an embodiment of the present application discloses a flow distribution obtaining method, which includes:
and S11, collecting flow information of an interface to be counted based on a tangent plane programming technology, storing the flow information into a flow distribution key value pair set, judging the content in the flow distribution key value pair set, and determining whether the current flow distribution key value pair set is in a statistically available state or not.
In this embodiment, firstly, a predefined annotation is added to the interface to be counted based on a tangent plane programming technique to obtain flow information of the interface to be counted. Firstly, the annotation is defined, so that the section programming technology is used, and the flow information can be collected only by annotating the interface needing to collect the flow distribution. After the traffic enters, it is necessary to determine the content of the traffic distribution map in the server. Where map is a set of key-value pairs (key-value), each element in the map set contains a key object and a value object. For storing data having a mapping relationship. Two sets of values are stored in the map set, one set of values is used to store the key in the map, the other set of values is used to store the value in the map, and the key and the value can be any reference type of data. The keys of the map are not allowed to repeat and the value may repeat, i.e. any two keys of the same map object are compared by the equal method and always returned to false. There is a one-way one-to-one relationship between keys and values in the map, i.e., unique, deterministic values are always found by the assigned keys. When the data is fetched from the map, the corresponding value can be fetched as long as the designated key is given. And judging the content in the flow distribution key value pair set, and determining whether the current flow distribution key value pair set is in a statistical state or not. If the current flow distribution key value pair set does not have flow distribution data in a preset historical time period, judging that the current flow distribution key value pair set is in a statistical state; if the number of the different values corresponding to the target field in the current flow distribution key value pair set does not exceed the preset threshold, judging that the current flow distribution key value pair set is in a statistical state. If the current flow distribution key value pair set has flow distribution data in a preset historical time period or the number of different values corresponding to the target fields in the current flow distribution key value pair set exceeds a preset threshold, corresponding fusing processing is performed to prohibit statistics on the values of all the target fields in the current flow distribution key value pair set. In a specific embodiment, if the traffic distribution data of the first two minutes (current minute-2) in the map is still (not written into the database), in order to not burden the memory of the server, a fusing process is required, i.e. the call is not counted. And in order to prevent the focus field (i.e. the target field) of the present minute from having too many different values, a threshold 5000 needs to be set, if the different values exceed this value, the fusing is also performed, and the statistics are not performed in the present call.
And step S12, if yes, counting the occurrence times of the flow distribution key value pair values of all the target fields in the set in a preset time period and updating the latest occurrence time.
In this embodiment, if the flow distribution map can count the flow in this minute, weight maintenance is performed on the value of the attention field that appears in the call, that is, the number of occurrences is increased, and the latest occurrence time when the value is received is updated.
Step S13, determining a ranking of the values of the target fields in the preset time period based on a preset rule, determining the occurrence number and the latest occurrence time of the values of the target fields corresponding to a target ranking based on the ranking, and deleting the values of the target fields after the target ranking according to the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking.
In this embodiment, a timing task is set for each server, and the current minute flow distribution map needs to be screened in a preset time period, for example, the current minute flow distribution map is screened every 10 seconds, so that the low-frequency flow is eliminated. Firstly, sorting the values of the target fields according to the preset timing screening time and the preset timing screening time based on the occurrence times of the values of the target fields in the preset time period, and if the occurrence times of the values of the target fields are the same, sorting the values of the target fields according to the latest occurrence time corresponding to the values of the target fields so as to determine the sorting of the values of the target fields. And deleting the value of the target field after the target ranking according to the occurrence number of the value of the target field corresponding to the target ranking and the latest occurrence time, and further comprising: comparing the occurrence times corresponding to the values of the target fields after the target ranking with the latest occurrence time, and obtaining a comparison result; deleting the value of the target field which accords with a preset deleting condition based on the comparison result, and not processing the value of the target field which does not accord with the preset deleting condition; the preset deleting condition is that the occurrence frequency corresponding to the value of the target field after the target ranking is smaller than the occurrence frequency of the value of the target field corresponding to the target ranking.
It is known that the value of the attention field is most likely to be high-frequency traffic if the number of cumulative occurrences is larger and the latest occurrence time is later (recently active), so that the elimination priority is set to be the first priority as the number of occurrences and the second priority as the latest occurrence time. And determining the sequence of the values of the target fields in the preset time period according to the occurrence times and the latest occurrence time, determining the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks based on the sequence, and deleting the values of the target fields after the target ranks according to the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks. Meanwhile, considering that the occurrence times and the latest occurrence time still change when the elimination mechanism is executed according to the priority order, when the elimination is performed, whether the elimination standard is met or not needs to be checked again, for example: when the first 1000 objects are reserved, when the objects corresponding to the values of the concerned fields after the 1000 objects are eliminated, the times of occurrence and the latest occurrence time of the objects and the 1000 objects need to be compared, and after the objects are still ranked at the 1000 objects, the objects are confirmed not to become high-frequency flow due to the rapid increase of the flow in the period of time. If the number of occurrences is greater than the number of occurrences of the sorted 1000, then the process is skipped and not performed.
Step S14, obtaining the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, in the flow distribution key value pair set within the preset statistical time, and storing the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, into a database.
In this embodiment, the value of the target field and the occurrence number of the value of the target field that meet the preset condition are assembled into an insert statement, so that the value of the target field and the occurrence number of the value of the target field that meet the preset condition are stored in a relational database management system based on the insert statement. In a specific embodiment, a timing task is set for each server, the top 100 value of the priority rank of each concerned field and the corresponding occurrence number are extracted from the traffic distribution map of the last minute every minute, and in order to reduce the interaction with the database, all concerned field traffic distributions of the database are assembled into a batch insert sql and stored in the database in batches. It should be noted that, in order to reduce the memory burden of the server, after the database is dropped, the dropped database data needs to be cleaned in time.
As can be seen from the above, when obtaining the flow distribution, the present application firstly collects the flow information of the interface to be counted based on the tangent plane programming technique, stores the flow information into the flow distribution key value pair set, judges the content in the flow distribution key value pair set, and determines whether the current flow distribution key value pair set is in a statistically available state; if yes, counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time; then determining the sequence of the values of the target fields in the preset time period based on a preset rule, determining the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks based on the sequence, and deleting the values of the target fields after the target ranks according to the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks; and finally, acquiring the value of the target field and the occurrence number of the value of the target field, which are in accordance with a preset condition, in the flow distribution key value pair set within a preset statistical time, and storing the value of the target field and the occurrence number of the value of the target field, which are in accordance with the preset condition, into a database. The application can be seen that the flow information is stored into the flow distribution key value pair set, when the flow distribution key value pair set is in a statistical state, the occurrence times of the values of all the target fields are counted and the latest occurrence time is updated, the values of all the target fields are sequenced after the counting and updating, so that the values of the target fields corresponding to the target ranks are determined, the values of the target fields after the target ranks can be deleted, finally, the values of all the target fields after the deletion are selected again, and the selected values of the target fields meeting the preset conditions and the occurrence times of the values of the target fields are stored into the database. Therefore, the application can compress the stored data to the minute dimension, only keep the head flow data needing to be concerned to the database, and facilitate the follow-up visual inquiry according to the time range and the multidimensional field of interest. And meanwhile, the storage cost is greatly reduced. As the stored information is reduced, the historical data does not need to be deleted, and the historical data can be checked at any time.
Referring to fig. 3, an embodiment of the present application discloses a specific flow distribution obtaining method, which includes:
in this embodiment, the flow distribution acquisition method disclosed in the present application specifically includes three modules: the system comprises a flow information collecting module, an elimination mechanism module and a storage module. The core of the flow collection module is to maintain the occurrence times and the latest occurrence time of different values of the concerned field, and firstly, judge the content of the flow distribution map in the server to judge whether the flow distribution map is in a collectable state. Taking the field of interest as a bold example, if the traffic distribution data of the first two minutes (the current minute-2) in the map is still (not written into the database), in order to not burden the memory of the server, a fusing process needs to be performed, that is, the current call is not counted. Because theoretically the traffic profile for the first two minutes should be written to the database by the memory module in the first minute. Meanwhile, in order to prevent the current minute from collecting different values of too many concerned fields, and aggravate the memory burden of the server, a threshold needs to be set, if the number of different values exceeds the threshold, the current call is fused, and statistics is not performed. If the server is in a collectable state, the occurrence number and the latest occurrence time of the current server in the current minute are maintained, namely, the occurrence number count of the bold is increased, and the latest occurrence time of the received bold is updated.
Then entering an elimination mechanism module, firstly traversing the current server flow distribution map, sorting the values of the concerned fields in the flow distribution map according to the occurrence times and the latest occurrence time, and recording the occurrence times and the latest occurrence time of the 1000 th bit, thereby removing the records of the bold after the 1000 th bit sorting, and meanwhile, the situation that the occurrence times and the latest occurrence time still change when the elimination mechanism is executed according to the priority sorting can be caused by the fact that the records after the 1000 th bit become high-frequency flow due to the rapid increase of the flow in the period of time. So the comparison is again made before removal, and if the number of occurrences after 1000 is greater than the number of occurrences of rank 1000 at this time, skip and do not process.
And finally, entering a storage module, namely acquiring the flow distribution map in the last minute after screening, and in order to reduce interaction with a database and improve the speed of writing into the database, assembling the flow distribution of all concerned fields in the database into batch inserters sql and storing the batch into the database because the core of the storage module is in efficiency and cost. I.e. to store bold, number of occurrences and machine ip (Internet Protocol ) to MySQL (relational database management system). In order to reduce the memory burden of the server, the data in the database needs to be cleaned in time after the database is dropped.
Therefore, the application can count approximate flow distribution under the condition of using a small amount of server resources and database storage to obtain the ranking condition of the flows with different values of the concerned field. Meanwhile, only head flow data needing to be concerned is reserved in the database, so that the storage cost is greatly reduced.
Referring to fig. 4, an embodiment of the present application discloses a flow distribution acquisition device, including:
the judging module 11 is configured to collect flow information of an interface to be counted based on a tangent plane programming technology, store the flow information into a flow distribution key value pair set, judge contents in the flow distribution key value pair set, and determine whether the current flow distribution key value pair set is in a statistically available state;
the statistics and updating module 12 is configured to, if yes, count the number of occurrences of the value of each target field in the set of the flow distribution key value pair in a preset period of time and update the latest occurrence time;
a deleting module 13, configured to determine an order of values of the target fields in the preset time period based on a preset rule, determine the occurrence number and the latest occurrence time of the values of the target fields corresponding to a target ranking based on the order, and delete the values of the target fields after the target ranking according to the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking;
the storage module 14 is configured to obtain the value of the target field and the number of occurrences of the value of the target field, which meet a preset condition, in the set of the flow distribution key value pairs within a preset statistical time, and store the value of the target field and the number of occurrences of the value of the target field, which meet the preset condition, in a database.
As can be seen from the above, when obtaining the flow distribution, the present application firstly collects the flow information of the interface to be counted based on the tangent plane programming technique, stores the flow information into the flow distribution key value pair set, judges the content in the flow distribution key value pair set, and determines whether the current flow distribution key value pair set is in a statistically available state; if yes, counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time; then determining the sequence of the values of the target fields in the preset time period based on a preset rule, determining the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks based on the sequence, and deleting the values of the target fields after the target ranks according to the occurrence times and the latest occurrence time of the values of the target fields corresponding to the target ranks; and finally, acquiring the value of the target field and the occurrence number of the value of the target field, which are in accordance with a preset condition, in the flow distribution key value pair set within a preset statistical time, and storing the value of the target field and the occurrence number of the value of the target field, which are in accordance with the preset condition, into a database. The application can be seen that the flow information is stored into the flow distribution key value pair set, when the flow distribution key value pair set is in a statistical state, the occurrence times of the values of all the target fields are counted and the latest occurrence time is updated, the values of all the target fields are sequenced after the counting and updating, so that the values of the target fields corresponding to the target ranks are determined, the values of the target fields after the target ranks can be deleted, finally, the values of all the target fields after the deletion are selected again, and the selected values of the target fields meeting the preset conditions and the occurrence times of the values of the target fields are stored into the database. Therefore, the application can compress the stored data to the minute dimension, only keep the head flow data needing to be concerned to the database, and facilitate the follow-up visual inquiry according to the time range and the multidimensional field of interest. And meanwhile, the storage cost is greatly reduced. As the stored information is reduced, the historical data does not need to be deleted, and the historical data can be checked at any time.
In some specific embodiments, the determining module 11 may specifically include:
and the flow information acquisition unit is used for adding a predefined annotation on the interface to be counted based on a section programming technology so as to acquire the flow information of the interface to be counted.
In some specific embodiments, the determining module 11 may specifically include:
the first judging unit is used for judging that the current flow distribution key value pair set is in a statistical state if the flow distribution data in the preset historical time period does not exist in the current flow distribution key value pair set;
and the second judging unit is used for judging that the current flow distribution key value pair set is in a statistical state if the number of different values corresponding to the target field in the current flow distribution key value pair set does not exceed a preset threshold value.
In some specific embodiments, the determining module 11 may further include:
and the statistics prohibiting unit is used for performing corresponding fusing processing to prohibit statistics on the values of all the target fields in the current flow distribution key value pair set if the flow distribution data in the preset historical time period exists in the current flow distribution key value pair set or the number of different values corresponding to the target fields in the current flow distribution key value pair set exceeds a preset threshold.
In some specific embodiments, the deletion module 13 may specifically include:
the sorting determining unit is configured to sort the values of the target fields according to a preset timing screening time based on the occurrence times of the values of the target fields in the preset time period, and if the occurrence times of the values of the target fields are the same, perform corresponding sorting according to the latest occurrence time corresponding to the values of the target fields, so as to determine the sorting of the values of the target fields.
In some specific embodiments, the deletion module 13 may further include:
a comparison result obtaining unit, configured to compare the occurrence number corresponding to the value of each target field after the target ranking with the latest occurrence time, and obtain a comparison result;
a deleting unit, configured to delete a value of the target field that meets a preset deleting condition based on the comparison result, and not process a value of the target field that does not meet the preset deleting condition; the preset deleting condition is that the occurrence frequency corresponding to the value of the target field after the target ranking is smaller than the occurrence frequency of the value of the target field corresponding to the target ranking.
In some specific embodiments, the storage module 14 may specifically include:
and the storage unit is used for assembling the value of the target field and the occurrence frequency of the value of the target field which meet the preset condition into an insert statement so as to store the value of the target field and the occurrence frequency of the value of the target field which meet the preset condition into a relational database management system based on the insert statement.
Further, the embodiment of the present application further discloses an electronic device, and fig. 5 is a block diagram of an electronic device 20 according to an exemplary embodiment, where the content of the figure is not to be considered as any limitation on the scope of use of the present application.
Fig. 5 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, which is loaded and executed by the processor 21 to implement relevant steps in the flow distribution acquisition method disclosed in any one of the foregoing embodiments. In addition, the electronic device 20 in the present embodiment may be specifically an electronic computer.
In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.
The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon may include an operating system 221, a computer program 222, and the like, and the storage may be temporary storage or permanent storage.
The operating system 221 is used for managing and controlling various hardware devices on the electronic device 20 and computer programs 222, which may be Windows Server, netware, unix, linux, etc. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the flow distribution acquisition method performed by the electronic device 20 as disclosed in any of the previous embodiments.
Further, the application also discloses a computer readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the previously disclosed flow distribution acquisition method. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing has outlined rather broadly the more detailed description of the application in order that the detailed description of the application that follows may be better understood, and in order that the present principles and embodiments may be better understood; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

Claims (10)

1. The flow distribution acquisition method is characterized by comprising the following steps of:
collecting flow information of an interface to be counted based on a tangent plane programming technology, storing the flow information into a flow distribution key value pair set, judging the content in the flow distribution key value pair set, and determining whether the current flow distribution key value pair set is in a statistically available state or not;
if yes, counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time;
determining the sequence of the values of the target fields in the preset time period based on a preset rule, determining the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking based on the sequence, and deleting the values of the target fields after the target ranking according to the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking;
and acquiring the value of the target field and the occurrence frequency of the value of the target field, which are in accordance with a preset condition, in the flow distribution key value pair set within a preset statistical time, and storing the value of the target field and the occurrence frequency of the value of the target field, which are in accordance with the preset condition, into a database.
2. The flow distribution obtaining method according to claim 1, wherein the collecting flow information of the interface to be counted based on the tangent plane programming technique includes:
and adding a predefined annotation on the interface to be counted based on a tangent plane programming technology so as to acquire the flow information of the interface to be counted.
3. The flow distribution acquisition method according to claim 1, wherein the determining whether the current flow distribution key value pair set is in a statistically acceptable state includes:
if the current flow distribution key value pair set does not have flow distribution data in a preset historical time period, judging that the current flow distribution key value pair set is in a statistical state;
if the number of the different values corresponding to the target field in the current flow distribution key value pair set does not exceed the preset threshold, judging that the current flow distribution key value pair set is in a statistical state.
4. The flow distribution acquisition method according to claim 3, wherein the determining whether the current flow distribution key value pair set is in a statistically acceptable state further comprises:
if the current flow distribution key value pair set has flow distribution data in a preset historical time period or the number of different values corresponding to the target fields in the current flow distribution key value pair set exceeds a preset threshold, corresponding fusing processing is performed to prohibit statistics on the values of all the target fields in the current flow distribution key value pair set.
5. The flow distribution acquisition method according to claim 1, wherein the determining the order of the values of the target fields in the preset time period based on a preset rule includes:
and sorting the values of the target fields according to the preset timing screening time based on the occurrence times of the values of the target fields in the preset time period, and if the occurrence times of the values of the target fields are the same, sorting the values of the target fields according to the latest occurrence time corresponding to the values of the target fields so as to determine the sorting of the values of the target fields.
6. The flow distribution acquisition method according to claim 1, characterized in that before deleting the value of the target field after the target ranking according to the number of occurrences of the value of the target field corresponding to the target ranking and the latest occurrence time, further comprising:
comparing the occurrence times corresponding to the values of the target fields after the target ranking with the latest occurrence time, and obtaining a comparison result;
deleting the value of the target field which accords with a preset deleting condition based on the comparison result, and not processing the value of the target field which does not accord with the preset deleting condition; the preset deleting condition is that the occurrence frequency corresponding to the value of the target field after the target ranking is smaller than the occurrence frequency of the value of the target field corresponding to the target ranking.
7. The flow distribution acquisition method according to any one of claims 1 to 6, characterized in that the storing the value of the target field and the number of occurrences of the value of the target field, which meet the preset condition, into a database includes:
and assembling the value of the target field and the occurrence frequency of the value of the target field which meet the preset condition into an insert statement, and storing the value of the target field and the occurrence frequency of the value of the target field which meet the preset condition into a relational database management system based on the insert statement.
8. A flow distribution acquisition device, comprising:
the judging module is used for collecting flow information of an interface to be counted based on a tangent plane programming technology, storing the flow information into a flow distribution key value pair set, judging the content in the flow distribution key value pair set and determining whether the current flow distribution key value pair set is in a statistically available state or not;
the statistics and updating module is used for counting the occurrence times of the flow distribution key value pair values of all target fields in the set in a preset time period and updating the latest occurrence time if the flow distribution key value pair values of all target fields are in the same time period;
a deleting module, configured to determine an order of values of the target fields in the preset time period based on a preset rule, determine the occurrence number and the latest occurrence time of the values of the target fields corresponding to a target ranking based on the order, and delete the values of the target fields after the target ranking according to the occurrence number and the latest occurrence time of the values of the target fields corresponding to the target ranking;
the storage module is used for acquiring the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, in the flow distribution key value pair set within the preset statistical time, and storing the value of the target field and the occurrence number of the value of the target field, which meet the preset condition, into a database.
9. An electronic device, comprising:
a memory for storing a computer program;
processor for executing the computer program to implement the steps of the flow distribution acquisition method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program; wherein the computer program when executed by a processor implements the steps of the flow distribution acquisition method according to any one of claims 1 to 7.
CN202310897129.6A 2023-07-20 2023-07-20 Flow distribution acquisition method, device, equipment and storage medium Pending CN116866213A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310897129.6A CN116866213A (en) 2023-07-20 2023-07-20 Flow distribution acquisition method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310897129.6A CN116866213A (en) 2023-07-20 2023-07-20 Flow distribution acquisition method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116866213A true CN116866213A (en) 2023-10-10

Family

ID=88233972

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310897129.6A Pending CN116866213A (en) 2023-07-20 2023-07-20 Flow distribution acquisition method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116866213A (en)

Similar Documents

Publication Publication Date Title
US20180365085A1 (en) Method and apparatus for monitoring client applications
US10108622B2 (en) Autonomic regulation of a volatile database table attribute
US9921936B2 (en) Method and system for IT resources performance analysis
US20050055673A1 (en) Automatic database diagnostic monitor architecture
JP2001043158A (en) Management data processor and computer-readable recording medium recorded with management data processing program
US9323805B2 (en) System and method for provenance function window optimization
CN102541884B (en) Method and device for database optimization
CN106383830B (en) Data retrieval method and equipment
CN110637292A (en) System and method for querying a resource cache
US10997171B2 (en) Database performance analysis based on a random archive
CN107330031B (en) Data storage method and device and electronic equipment
CN110543509B (en) Monitoring system, method and device for user access data and electronic equipment
CN116700624A (en) Data deleting method and device, electronic equipment and storage medium
CN116866213A (en) Flow distribution acquisition method, device, equipment and storage medium
Tao et al. Logging every footstep: Quantile summaries for the entire history
CN113377519B (en) CDN-based content scheduling method
CN115269519A (en) Log detection method and device and electronic equipment
CN114827951A (en) Vehicle network quality analysis method and system based on vehicle terminal and storage medium
CN111858733A (en) Government affair information comparison method and system based on internet multi-source heterogeneous data
CN109189743B (en) Super node recognition filtering method and system with low resource consumption and oriented to large-flow real-time graph data
JP3742378B2 (en) Entrance / exit history record management device
CN116166647A (en) Data management method and device and electronic equipment
CN111222897B (en) Client Internet surfing satisfaction prediction method and device
CN115934782B (en) Method for analyzing and processing security log and computer storage medium
CN111737281B (en) Database query method, device, electronic equipment and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20231109

Address after: 200441 Room B1-7001, Building 3, No. 258 Changjiang Road, Baoshan District, Shanghai

Applicant after: Shanghai Mengju Information Technology Co.,Ltd.

Address before: Room b1-8008, building 3, 258 Changjiang Road, Baoshan District, Shanghai 200441

Applicant before: Shanghai Weimeng Enterprise Development Co.,Ltd.

TA01 Transfer of patent application right