CN115314423A - Traffic data statistical method, device and storage medium - Google Patents
Traffic data statistical method, device and storage medium Download PDFInfo
- Publication number
- CN115314423A CN115314423A CN202210846343.4A CN202210846343A CN115314423A CN 115314423 A CN115314423 A CN 115314423A CN 202210846343 A CN202210846343 A CN 202210846343A CN 115314423 A CN115314423 A CN 115314423A
- Authority
- CN
- China
- Prior art keywords
- statistical
- flow
- data
- flow data
- traffic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000007619 statistical method Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 30
- 238000013523 data management Methods 0.000 claims abstract description 24
- 238000004590 computer program Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 2
- 230000015654 memory Effects 0.000 description 12
- 230000005540 biological transmission Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 235000019580 granularity Nutrition 0.000 description 6
- 238000007726 management method Methods 0.000 description 6
- 238000009825 accumulation Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 238000012216 screening Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 108010001267 Protein Subunits Proteins 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
Landscapes
- Engineering & Computer Science (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application relates to a traffic data statistical method, a device and a storage medium, wherein the method is applied to a traffic data management system, and the system comprises a traffic database, cache middleware and a task statistical engine. The method comprises the following steps: receiving a statistic task of flow data; counting the flow data in the flow database based on the counting task to obtain first flow data; storing the first flow data into a cache middleware; extracting corresponding first flow data from the caching middleware and displaying the first flow data based on the query request; the flow database does not need to be accessed for statistics according to the query request, but the corresponding result is directly extracted from the cache middleware, so that the efficiency of querying and displaying the flow data is improved, and the problem that the traditional database is low in efficiency in the aspect of flow data statistics of the Internet of things is solved.
Description
Technical Field
The application relates to the technical field of internet of things, in particular to a traffic data statistical method, a traffic data statistical device and a storage medium.
Background
The traffic data of the internet of things generally includes parameters such as network protocol type, traffic direction, device network information, traffic generation time and the like. The flow data statistics can reflect the flow transmission condition of specific time or specific equipment, and is used for flow management personnel to check so as to evaluate the information transmission quantity and judge whether the flow transmission is abnormal. For a single internet of things device, the traffic data volume is not outstanding, but with the increase of the access number of the service devices of the internet of things system, the traffic data rises sharply, and more storage space and better retrieval statistical performance are needed. For general data storage, a scheme that a traditional database or a big data storage system is used is generally adopted, and a statistical analysis capability of a third-party storage system is utilized to provide services when a query is made. But the method is limited by resources (hard disks, memories, networks and the like) of the internet of things equipment, a large data storage system is difficult to apply, and meanwhile, the performance and statistical efficiency of a traditional database in the aspect of query and statistics of massive internet of things flow data are low, so that the user requirements are difficult to meet.
Aiming at the problem that the traditional database in the related technology is low in efficiency in traffic data statistics of the Internet of things, no effective solution is provided at present.
Disclosure of Invention
The embodiment provides a traffic data statistical method, a traffic data statistical device and a storage medium, so as to solve the problem that the performance of a traditional database in the traffic data statistics aspect of the internet of things is poor in the related art.
In a first aspect, in this embodiment, a traffic data statistics method is provided, and is applied to a traffic data management system, where the traffic data management system includes a traffic database, caching middleware, and a task statistics engine, and the method includes:
receiving a statistic task of flow data;
counting the traffic data in the traffic database based on the counting task to obtain first traffic data;
storing the first traffic data into the caching middleware;
and extracting and displaying corresponding first flow data from the cache middleware based on the query request.
In some of these embodiments, the statistical task includes at least one statistical parameter value, and the performing statistics on the flow data in the flow database based on the statistical task includes:
querying the flow database based on the at least one statistical parameter value to obtain second flow data;
summing the flow values in the second flow data to obtain a statistical flow value;
and generating the first flow data according to a preset format based on the statistical flow value and the statistical parameter value.
In some of these embodiments, the generating the first flow data in a preset format based on the statistical flow value and the statistical parameter value comprises:
setting the statistical parameter value as a Key value of the first flow data;
and setting the statistical flow Value as a Value of the first flow data.
In some embodiments, said querying said traffic database based on said at least one statistical parameter value, and obtaining second traffic data comprises:
converting the statistical parameter values into corresponding database query conditions;
splicing the database query conditions corresponding to the at least one statistical parameter value into a database query statement;
and querying the flow database through the database query statement to acquire second flow data.
In some embodiments, the extracting and displaying, based on the query request, the corresponding first traffic data from the caching middleware includes:
acquiring query parameters in the query request;
querying a Key value of the first flow data based on the query parameter;
and displaying the statistical flow value in the first flow data under the condition that the Key value is consistent with the query parameter.
In some of these embodiments, the query parameter includes a flow generation time, and the displaying the statistical flow value in the first flow data if the Key value is consistent with the query parameter includes:
determining a difference between a query time and the traffic generation time;
and acquiring corresponding first flow data based on the difference value.
In some embodiments, the statistical task includes at least one statistical parameter value, the statistical parameter value includes a statistical period, and the performing statistics on the flow data in the flow database based on the statistical task includes:
determining the execution time of the statistical task based on the statistical period;
and counting the flow data generated in the counting period in the execution time, and acquiring first flow data corresponding to the counting period.
In some of these embodiments, after storing the first traffic data in caching middleware, the method further comprises:
setting a corresponding expiration time for the first stream data;
deleting the first stream data if a storage time of the first stream data in the caching middleware exceeds the expiration time.
In a second aspect, in this embodiment, there is provided a traffic data statistics apparatus applied to a traffic data management system, where the traffic data management system includes a traffic database, caching middleware, and a task statistics engine, the apparatus includes:
the receiving module is used for receiving a statistical task of the flow data;
the statistical module is used for carrying out statistics on the flow data in the flow database based on the statistical task to obtain first flow data;
a storing module, configured to store the first traffic data into the caching middleware;
and the extracting module is used for extracting and displaying corresponding first flow data from the cache middleware based on the query request.
In a third aspect, in the present embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the traffic data statistical method according to the first aspect.
Compared with the related art, the traffic data statistical method provided in this embodiment determines statistical content and statistical dimensions corresponding to user requirements through a statistical task of receiving traffic data; counting the flow data in the flow database based on the counting task and acquiring first flow data, screening corresponding data in the flow database according to user requirements and carrying out statistical calculation according to a certain statistical dimension to acquire a statistical result; the first flow data is stored in the cache middleware, and the statistical result is temporarily stored in the cache middleware without being stored in a hard disk, so that the occupation of hard disk resources is reduced; by extracting and displaying the corresponding first flow data from the cache middleware based on the query request, the corresponding result is directly extracted from the cache middleware without accessing the flow database for statistics according to the query request, the efficiency of querying and displaying the flow data is improved, and the problem that the traditional database is low in efficiency in the aspect of flow data statistics of the Internet of things is solved.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic diagram of a traffic data management system to which a traffic data statistical method according to an embodiment of the present application is applied;
FIG. 2 is a flow chart of a traffic data statistics method according to an embodiment of the present application;
FIG. 3 is a flowchart of acquiring first traffic data according to an embodiment of the present application;
fig. 4 is a flowchart of extracting first traffic data from caching middleware according to an embodiment of the present application;
FIG. 5 is a flow chart of a traffic data statistics method of the preferred embodiment of the present application;
fig. 6 is a block diagram of a flow rate data statistics device according to an embodiment of the present application.
Detailed Description
For a clearer understanding of the objects, aspects and advantages of the present application, reference is made to the following description and accompanying drawings.
Unless defined otherwise, technical or scientific terms used herein shall have the same general meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The use of the terms "a" and "an" and "the" and similar referents in the context of this application do not denote a limitation of quantity, either in the singular or the plural. The terms "comprises," "comprising," "has," "having," and any variations thereof, as referred to in this application, are intended to cover non-exclusive inclusions; for example, a process, method, and system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or modules, but may include other steps or modules (elements) not listed or inherent to such process, method, article, or apparatus. Reference in this application to "connected," "coupled," and the like is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference to "a plurality" in this application means two or more. "and/or" describes the association relationship of the associated object, indicating that there may be three relationships, for example, "a and/or B" may indicate: a exists alone, A and B exist simultaneously, and B exists alone. In general, the character "/" indicates a relationship in which the objects associated before and after are an "or". Reference in the present application to the terms "first," "second," "third," etc., merely distinguish between similar objects and do not denote a particular order or importance to the objects.
The method embodiment provided in this embodiment may be applied to a traffic data management system, and may specifically be executed in a control unit of the traffic data management system, where the control unit may be a processor of the traffic data management system. The control unit may comprise one or more processors and a memory for storing data, wherein the processors may comprise, but are not limited to, processing means such as a microprocessor MCU or a programmable logic device FPGA. The control unit may further include a transmission device for a communication function and an input-output device, may communicate with a remote server through a network, and may perform data processing and storage through the remote server.
The memory can be used for storing computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the traffic data statistical method in the embodiment, and the processor executes various functional applications and data processing by running the computer program stored in the memory, that is, implementing the method described above. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, the memory may further include memory remotely located from the processor, and these remote memories may be connected to the traffic data management system over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device is used to receive or transmit data via a network. The network described above includes a wireless network provided by a communication provider of the traffic data management system. In one example, the transmission device includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
Fig. 1 is a schematic diagram of a traffic data management system to which the traffic data statistical method of the present embodiment is applied. As shown in fig. 1, the system of the present embodiment includes a traffic database 12, a caching middleware 13, and a task statistics engine 11. The traffic database 12 may be a relational database such as SQL database or the like for storing traffic data, which may be generated based on traffic generated in real time and may include information such as traffic size, traffic direction, IP address, application protocol, and time of occurrence. The cache middleware 13 may be a memory-based database, such as Redis. The cache middleware 13 is configured to store statistical results of the flow data, and provide corresponding query results for display when a user sends a query request. The task statistics engine 11 is a main body for executing the traffic statistics task, and after receiving the statistics task, accesses the traffic database according to the content of the statistics task to obtain corresponding traffic data, performs statistics to obtain a statistical result, and stores the statistical result in the caching middleware 13. It will be understood by those of ordinary skill in the art that the structure shown in fig. 1 is merely an illustration and is not intended to limit the structure of the traffic data management system. For example, the traffic data management system may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
In this embodiment, a traffic data statistical method is provided, and fig. 2 is a flowchart of the traffic data statistical method of this embodiment, as shown in fig. 2, the flowchart includes the following steps:
step S201, a statistical task of the traffic data is received.
In an application scenario of traffic data statistics, a statistical task may include multiple statistical dimensions such as a traffic size, a traffic direction, a traffic access address, a traffic generation time, and a network protocol, and each statistical task may be formed by combining multiple statistical dimensions. For example, one statistical task may be to count traffic generation time from 8 am 8 pm to 16 pm on 1 st 6 th 3 rd 3 th 3 rd 2022, the traffic is in the uplink direction, and the network protocol is the total traffic value of HTTP.
Step S202, flow data in the flow database are counted based on the counting task, and first flow data are obtained.
The traffic data in the traffic database may be data recorded in real time according to the time when the traffic occurs, and the recorded dimensions also include the traffic size, the traffic direction, the traffic access address, the traffic generation time, the network protocol, and the like. And finding data meeting statistical conditions in the database according to the statistical task, and performing statistics according to the requirements of statistical figures to obtain first flow data.
Step S203, storing the first flow data into the caching middleware.
And step S204, extracting corresponding first flow data from the caching middleware and displaying the first flow data based on the query request.
The query request may select and display the first traffic data in the caching middleware based on one or more statistical dimensions. For example, the uplink flow rate counted in days from 1/6/10/2022 year may be selected and extracted and displayed by a line graph or a bar graph.
Through the steps S201 to S204, the statistical content and the statistical dimension corresponding to the user requirement are determined through the statistical task of receiving the flow data; counting the flow data in the flow database based on the counting task and acquiring first flow data, screening corresponding data in the flow database according to user requirements and carrying out statistical calculation according to a certain statistical dimension to acquire a statistical result; the first flow data is stored in the cache middleware, and the statistical result is temporarily stored in the cache middleware without being stored in the hard disk, so that the occupation of hard disk resources is reduced; by extracting and displaying the corresponding first flow data from the cache middleware based on the query request, the corresponding result is directly extracted from the cache middleware without accessing the flow database for statistics according to the query request, the efficiency of querying and displaying the flow data is improved, and the problem that the traditional database is low in efficiency in the aspect of flow data statistics of the Internet of things is solved.
In some of these embodiments, the statistical task includes at least one statistical parameter value. Fig. 3 is a flowchart of obtaining first flow data by performing statistics on flow data based on a statistical task according to this embodiment, and as shown in fig. 3, the flowchart includes the following steps:
step S301, inquiring a flow database based on at least one statistical parameter value to obtain second flow data.
The statistical parameter values are corresponding values of the statistical dimensions. For example, for a statistical dimension of flow generation time, the statistical parameter value is a specific date and time value. For the statistical dimension of the flow direction, the statistical parameter value is uplink or downlink. When a plurality of statistical parameter values are included in one statistical task, the obtained second flow data should satisfy the requirement of each statistical parameter value.
In some embodiments, the second traffic data may also be obtained based on a threshold range of the statistical parameter value. For example, when the statistical parameter value generates a time value for a specified flow rate, the second flow data may be flow data that occurs a period of time before or after the specified time value. For example, when the statistical parameter value is 11 am 15 pm on 13 d 6/2022, the second flow rate data is the flow rate data between 11 pm and 11 am 15 pm on that day. The threshold range may be set in the system in advance or set in the statistical task.
Step S302, summing the flow values in the second flow data, and obtaining a statistical flow value.
And summing the second flow data meeting the requirement of each statistical parameter value to obtain a statistical flow value, namely the statistical flow value corresponding to the statistical task.
Step S303, based on the statistical flow value and the statistical parameter value, generating first flow data according to a preset format.
And combining the statistical flow value with the statistical parameter value in the statistical task, and storing the combined value in the caching middleware according to a preset format. The preset format comprises field setting of a database, key/Value setting, a combination mode of a statistical flow Value and a statistical parameter Value, symbol combination setting, a storage format and the like, and any one or more formats can be set.
Through the steps S301-S303, acquiring second flow data by querying a flow database based on the statistical parameter values, and screening the flow database to acquire flow data corresponding to the statistical task; obtaining a statistical flow value by summing the flow values in the second flow data to obtain a statistical result; the first flow data is generated according to the statistical flow value and the statistical parameter value and the preset format, the storage format of the first flow data is unified, the storage space utilization rate and the read-write efficiency are improved, and the query efficiency of the first flow data is improved.
In some of these embodiments, a detailed procedure is involved for generating first traffic data in a preset format based on statistical traffic values and statistical parameter values. The process comprises the following steps:
step S11, setting the statistical parameter value as a Key value of the first flow data;
and S12, setting the statistical flow Value as the Value of the first flow data.
Based on the characteristic that the flow data query needs more, the cache middleware can be a Key-Value database. Based on query requirements, at least one statistical dimension of the flow data can be used as a Key format component element, and a statistical result is used as a Value format component element, so that the data query efficiency is improved. Correspondingly, when the first traffic data is stored in the cache middleware, the data can be correspondingly stored according to a preset Key format and a preset Value format, a statistical parameter Value corresponding to a statistical dimension in the Key format is set as a Key Value, and a statistical traffic Value corresponding to a statistical result in the Value format is set as a Value. In some embodiments, other statistical dimensions besides listed Key format may also be used as elements of the Value format.
Based on different query and statistical requirements, the first traffic data can be stored by using different Key-Value formats, corresponding to different Key-Value databases.
Through the steps S11 to S12, the statistical parameter Value is set as the Key Value of the first flow data, the statistical flow Value is set as the Value of the first flow data, the statistical parameter Value and the statistical flow Value are associated through the storage format of the Key-Value database, and when the query request is queried through the statistical dimension specified by the Key format, the query result acquisition time is shortened, and the query efficiency is improved.
In some embodiments, a common statistical parameter value may also be used to assign a value to a statistical dimension in the query requirement, set the value as a Key format of the first traffic data, and set the value and the statistical task in a unified manner, so as to further improve the query efficiency.
For example, the statistical dimensions of the statistical task include a traffic direction, a traffic generation time, a traffic accumulation time, and a network protocol, the traffic accumulation time may be set to 30 minutes, the traffic direction may be set to an uplink, the statistical task may be set in combination with a specific traffic generation time, and corresponding first traffic data may be acquired. Based on the setting, the Key format of the cache middleware is set to be ' Flow: half: up: { time } ' or ' Flow: half: down: { time } ', the Value format is set to be ' { ' protocol ': flow size ', \ 8230 } ', namely, the constituent elements in the Key format are Flow direction, flow generation time and Flow accumulation time, half represents that the Flow accumulation time is 30 minutes, up represents that the Flow direction is uplink, and time represents the Flow generation time. The components of the Value format include a statistical traffic Value and a network protocol. The first flow data corresponding to the Key/Value format of the database represents the statistical flow values with a certain time as the midpoint time, and the flow direction is the upward direction, and the statistics flow values are counted backwards/forwards for 15 minutes.
And after the first flow data is acquired by the statistical task, storing the first flow data according to the format. For example, the Key Value of the first Flow data is "Flow: half: up:202207131115", the Value is "{" FTP ": 23.5", "HTTP": 56.3"}", which means that 15-time at 13/7/2022 is divided into midpoint time, statistics occur between 11/30/11/day, and the Flow direction is an upstream Flow Value, wherein the Flow Value transmitted through the FTP protocol is 23.5, and the Flow Value transmitted through the HTTP protocol is 56.3. Therefore, when the query request needs to acquire the flow statistical data with 30-minute period and ascending flow direction at 7, 13 and 2022, the flow statistical data can be directly acquired from the first flow data and displayed, further statistics on the first flow data is not needed, and the query and display efficiency is remarkably improved.
In some embodiments, the specific process of querying the traffic database to obtain the second traffic data based on the statistical parameter value includes the following steps:
step S21, converting the statistical parameter values into corresponding database query conditions;
s22, splicing the database query conditions corresponding to at least one statistical parameter value into a database query statement;
and step S23, inquiring the flow database through the database inquiry statement to acquire second flow data.
The flow database may be a relational database such as an SQL database, and the statistical parameter values are converted into corresponding SQL query statements for querying. When the number of the statistical parameter values is multiple, the query conditions corresponding to each statistical parameter value can be spliced into one SQL statement, and the flow database is queried through the SQL statement to obtain second flow data.
Through the steps S21-S23, the statistical parameter values are converted into the corresponding database query conditions, one or more database query conditions are spliced into one database query statement for query, the statistical requirements are converted into the corresponding database query statement, the acquisition speed of the second flow data is increased, and the query efficiency is improved.
In some embodiments, fig. 4 is a flowchart of the present embodiment, where the corresponding first traffic data is extracted from the cache middleware based on the query request and displayed, and as shown in fig. 4, the flowchart includes the following steps:
step S401, obtaining query parameters in the query request;
step S402, inquiring a Key value of the first flow data based on the inquiry parameters;
and step S403, displaying the statistical flow value in the first flow data under the condition that the Key value is consistent with the query parameter.
The query request includes at least one query parameter, which may correspond to a Key format of a Key-Value database. For example, the Key format of the Key-Value database includes statistical dimensions such as a traffic IP address and traffic generation time, and the query parameter also includes the traffic IP address, so that the query request corresponds to the Key-Value database, and the first traffic data in the Key-Value database can be obtained through the query parameter. And when the query parameter is consistent with the Key Value of the first flow data in the Key-Value database, the query result meets the query request, and the result is displayed.
For example, when the query request is to query a traffic Value of an access IP address within a certain time period, the query parameter includes two statistical dimensions of the traffic IP address and the traffic generation time, and the two statistical dimensions are consistent with a Key format of a Key-Value database of the caching middleware, so that corresponding data can be obtained by querying the database. And searching first flow data with the Key value as the specific IP address and the flow generation time in the specific time period in the database, and displaying the first flow data meeting the conditions.
Through the steps S401 to S403, the corresponding database is found by acquiring the query parameters in the query request and comparing the query parameters with the Key format of the database; and comparing the query parameters with the Key values in the database to determine corresponding first flow data, so as to realize the query function of the flow data.
In some embodiments, the query parameter includes a flow generation time, and in a case where the Key value is consistent with the query parameter, the process of displaying the statistical flow value in the first flow data includes the following steps:
step S31, determining the difference value between the query time and the flow generation time;
the flow generation time refers to a generation time or a time period corresponding to the first flow data, and the query time refers to a time when the query request is generated. Due to the limited space of the caching middleware, a large amount of flow data cannot be stored for a long time. Therefore, the traffic data management system can store traffic data with longer storage time in a larger granularity. For example, when the storage time is short, the flow data may be counted and stored with 30 minutes as one counting period; when the storage time is long, the traffic data can be counted and stored in a counting period of 24 hours. Both data may be stored in the same database but represented with different characters in the Key format to distinguish upon query.
When a query request needs to query traffic data far away from the current time, a corresponding Key format is determined according to a difference value between the query time and traffic generation time, for example, if the difference value exceeds 24 hours, the corresponding Key format includes a "Day" character, and if the difference value is smaller than 24 hours, the corresponding Key format includes a "Half" character.
And step S32, acquiring corresponding first flow data based on the difference value.
And inquiring according to characters in a Key format to obtain corresponding first flow data.
Through the steps S31 to S32, the difference Value between the query time and the flow generation time is determined, different difference values are distinguished through characters in a Key format, the corresponding first flow data is obtained through query according to the characters, the data with different storage times are classified and queried through a faster Key-Value mode, the storage time of the first flow data does not need to be calculated, the query result is determined according to the calculation result, and the query efficiency is improved.
In some embodiments, the statistical task includes at least one statistical parameter value, the statistical parameter value includes a statistical period, the flow data in the flow database is counted based on the statistical task, and the process of obtaining the first flow data includes the following steps:
s41, determining the execution time of the statistical task based on the statistical period;
some statistical tasks are timing statistics, such as counting cumulative flow values per day or per hour. For the statistical task with regular interval time, the execution time can be set by the parameter of the statistical period. For example, for a traffic statistic task performed once a day from 19 o ' clock before the day to 19 o ' clock at the day, the execution time of the traffic statistic task may be determined to be 19 o ' clock 05 minutes per day based on a preset interval time.
And S42, counting the flow data generated in the counting period in the execution time, and acquiring first flow data corresponding to the counting period.
The task counting engine can start a counting task according to the execution time, obtain first flow data corresponding to the counting period and store the first flow data in the cache middleware. And closing the task after the counting task is completed.
Through the steps S41 to S42, the execution time of the statistical task is determined based on the statistical period, the first flow data generated in the statistical period is counted at the execution time, the execution time can be automatically set and the periodic statistical task can be automatically executed, the user does not need to issue the statistical task for multiple times, timing and automation of flow statistics are realized, and the statistical efficiency is improved.
In some embodiments, after storing the first traffic data in the caching middleware, the flow of the traffic data statistical method of the embodiment includes the following steps:
s51, setting corresponding expiration time for the first stream data;
and S52, deleting the first flow data when the storage time of the first flow data in the cache middleware exceeds the expiration time.
Due to the storage space limitation of the caching middleware, the storage space can be saved by setting different expiration times for the first stream data with different statistical granularities. When the data is out of date, the corresponding first flow data can be deleted to make up space for storing new flow data. The finer the statistical granularity, the more the first traffic data, the shorter the expiration time. For example, the expiration time may be set to 24 hours and 20 minutes for the first traffic data with a statistical granularity of 30 minutes, and 30 days for the first traffic data with a statistical granularity of 24 hours.
Through the steps S51 to S52, the corresponding expiration time is set for the first traffic data, and the first traffic data is deleted when the storage time exceeds the expiration time, so that the storage space of the cache middleware is saved, and the long-term query requirement of the user on the first traffic data is met.
The present embodiment is described and illustrated below by means of preferred embodiments.
The traffic data statistical method of the preferred embodiment is applied to a traffic data management system, and the traffic data management system comprises a front-end interface, a task statistical management module, a data statistical result management module, a task statistical engine, a cache middleware and a traffic database.
And after the flow data is accessed, the flow data is stored in a flow database through a normal storage program. The front-end interface is used for configuring the statistical task. The caching middleware is Redis.
The task statistics management module is used for receiving operation instructions (addition, deletion, modification and check) of a front-end interface, converting the operation instructions into operation on a flow database, and simultaneously issuing corresponding commands (starting and stopping) to the task statistics engine; the front-end interface can select different statistical scales according to business requirements.
The data statistical result management module is used for providing a statistical result query interface and managing data statistical results, including query and deletion of the data statistical results.
The task statistics engine is a core module of the whole flow data management system and is responsible for receiving the statistics tasks issued by the task statistics management module and starting or closing the statistics tasks; and the lower part is responsible for pulling original flow data from the flow database according to the content of the statistical task, realizing timing statistics according to the statistical scale, and finally storing the result into the cache middleware.
Fig. 5 is a flowchart of the traffic data statistical method according to the preferred embodiment. As shown in fig. 5, the process includes the following steps:
s501, receiving a statistic task of flow data, wherein statistic dimensions in the statistic task comprise a statistic period, flow generation time, a flow direction and an application protocol;
the flow direction comprises: displaying the uplink flow by default according to the uplink flow, the downlink flow and the total flow;
the flow statistic period comprises: the last three hours, the last day, the last week, the last month, the last three hours are displayed by default;
the application protocol comprises the following steps: application layer protocols such as HTTP, FTP, etc.
S502, determining the execution time of the statistical task based on the statistical period;
s503, counting the flow data generated in the counting period in the execution time;
s504, inquiring a flow database through a database query statement corresponding to the statistical parameter value to obtain second flow data;
s505, summing the flow values in the second flow data to obtain a statistical flow value;
s506, generating a Key format of first flow data based on flow generation time, flow direction and statistical period; generating a Value format of the first flow data based on the statistical flow Value and the application protocol, and storing the first flow data according to the Key/Value format;
the storage structure of the cache middleware may include:
a. and (3) whole network flow and application flow statistics:
b. destination IP flow statistics:
s507, storing the first flow data into a cache middleware;
s508, setting corresponding expiration time for the first stream data;
s509, deleting the first traffic data when the storage time of the first traffic data in the caching middleware exceeds the expiration time;
s510, obtaining query parameters in the query request, wherein the query parameters comprise traffic generation time;
s511, determining the difference value between the query time and the flow generation time;
s512, acquiring corresponding first flow data based on the difference value;
s513, querying a Key value of the first flow data based on other query parameters;
and S514, displaying the statistical flow value in the first flow data under the condition that the Key value is consistent with the query parameter.
Through the steps S501 to S514, determining the statistical dimension, the statistical parameter value and the task execution time corresponding to the user requirement, executing flow statistics according to the execution time, converting the statistical parameter value into a corresponding database query statement, acquiring second flow data, and performing statistics to acquire a statistical flow value; the statistical parameter values and the statistical flow values are combined according to a Key/Value format to generate first flow data, the first flow data are stored in a cache middleware, the cache middleware is used for temporarily storing statistical results, the statistical results do not need to be stored in a hard disk, and the occupation of hard disk resources is reduced; the query request is compared with the statistical parameter values in the Key format to extract and display corresponding first flow data, and the query efficiency of the flow data is improved by utilizing the high-efficiency query performance of the Key/Value database; the flow database does not need to be accessed for statistics in each query, but the corresponding result is directly extracted from the cache middleware, so that the efficiency of querying and displaying the flow data is improved, and the problem that the traditional database is low in efficiency in the aspect of flow data statistics of the Internet of things is solved; by setting corresponding expiration time for the first flow data, the storage time classification is carried out on the flow data with different statistical granularities, and the space utilization efficiency of the cache middleware is improved.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
In some embodiments, the present application further provides a traffic data statistics apparatus, which is applied to a traffic data management system, where the traffic data management system includes a traffic database, a caching middleware, and a task statistics engine. The flow data statistics apparatus is used to implement the above embodiments and preferred embodiments, and the description thereof is omitted for brevity. The terms "module," "unit," "sub-unit," and the like as used below may implement a combination of software and/or hardware of predetermined functions.
In some embodiments, fig. 6 is a block diagram of a flow data statistics apparatus of the present embodiment, and as shown in fig. 6, the apparatus includes:
a receiving module 61, configured to receive a statistical task of traffic data;
the statistical module 62 is configured to perform statistics on the traffic data in the traffic database based on the statistical task to obtain first traffic data;
a storing module 63, configured to store the first traffic data into the caching middleware;
and the extracting module 64 is configured to extract and display corresponding first traffic data from the caching middleware based on the query request.
The traffic data statistical apparatus in this embodiment receives a statistical task of traffic data through the receiving module 61, and determines statistical content and statistical dimensions corresponding to user requirements; counting the flow data in the flow database and acquiring first flow data based on the counting task through a counting module 62, screening corresponding data in the flow database according to user requirements and performing statistical calculation according to a certain statistical dimension to acquire a statistical result; the first flow data is stored in the cache middleware through the storage module 63, the statistical result is temporarily stored through the cache middleware, the statistical result does not need to be stored in a hard disk, and the occupation of hard disk resources is reduced; the extraction module 64 extracts and displays the corresponding first flow data from the cache middleware based on the query request, and directly extracts the corresponding result from the cache middleware without accessing the flow database for statistics according to the query request, so that the efficiency of querying and displaying the flow data is improved, and the problem of low efficiency of the traditional database in the aspect of flow data statistics of the internet of things is solved.
In some embodiments, the statistical task includes at least one statistical parameter value, the statistical module includes a first obtaining sub-module, a second obtaining sub-module, and a generating sub-module, the first obtaining sub-module is configured to query the traffic database based on the at least one statistical parameter value to obtain second traffic data; the second obtaining submodule is used for summing the flow values in the second flow data to obtain a statistical flow value; the generation submodule is used for generating first flow data according to a preset format based on the statistical flow value and the statistical parameter value.
In the traffic data statistical apparatus provided in this embodiment, the first obtaining sub-module queries the traffic database based on the statistical parameter value to obtain the second traffic data, and screens the traffic data corresponding to the statistical task from the traffic database; summing the flow values in the second flow data through a second obtaining submodule to obtain a statistical flow value, and obtaining a statistical result; the generation submodule generates the first flow data according to the statistical flow value and the statistical parameter value and the preset format, unifies the storage format of the first flow data, improves the utilization rate of the storage space and the read-write efficiency, and improves the query efficiency of the first flow data.
In some embodiments, the generation submodule further includes a first setting unit and a second setting unit, and the first setting unit is configured to set the statistical parameter value as a Key value of the first traffic data; the second setting unit is used for setting the statistical flow Value as the Value of the first flow data.
In the traffic data statistics apparatus provided in this embodiment, the first setting unit sets the statistical parameter Value as a Key Value of the first traffic data, the second setting unit sets the statistical traffic Value as a Value of the first traffic data, and the storage format of the Key-Value database associates the statistical parameter Value with the statistical traffic Value.
In some embodiments, the first obtaining sub-module further includes a conversion unit, a splicing unit, and a query unit, where the conversion unit is configured to convert the statistical parameter value into a corresponding database query condition; the splicing unit is used for splicing the database query conditions corresponding to at least one statistical parameter value into database query statements; the query unit is used for querying the flow database through the database query statement to acquire second flow data.
The traffic data statistical device provided by this embodiment converts the statistical parameter values into corresponding database query conditions through the conversion unit, splices one or more database query conditions into a database query statement through the splicing unit and the query unit to query, converts the statistical requirements into corresponding database query statements, improves the acquisition speed of the second traffic data, and improves the query efficiency.
In addition, in combination with the traffic data statistical method provided in the foregoing embodiment, a storage medium may also be provided in this embodiment. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the traffic data statistics methods of the above embodiments.
It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementations, and details are not described again in this embodiment.
It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to be limiting. All other embodiments, which can be derived by a person skilled in the art from the examples provided herein without any inventive step, shall fall within the scope of protection of the present application.
It is obvious that the drawings are only examples or embodiments of the present application, and it is obvious to those skilled in the art that the present application can be applied to other similar cases according to the drawings without creative efforts. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
The term "embodiment" is used herein to mean that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly or implicitly understood by one of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the patent protection. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A traffic data statistical method is applied to a traffic data management system, and is characterized in that the traffic data management system comprises a traffic database, cache middleware and a task statistical engine, and the method comprises the following steps:
receiving a statistic task of flow data;
counting the flow data in the flow database based on the counting task to obtain first flow data;
storing the first traffic data into the caching middleware;
and extracting and displaying corresponding first flow data from the cache middleware based on the query request.
2. The method of claim 1, wherein the statistical task comprises at least one statistical parameter value, wherein performing statistics on the flow data in the flow database based on the statistical task, and wherein obtaining the first flow data comprises:
querying the flow database based on the at least one statistical parameter value to obtain second flow data;
summing the flow values in the second flow data to obtain a statistical flow value;
and generating the first flow data according to a preset format based on the statistical flow value and the statistical parameter value.
3. The method of claim 2, wherein generating the first flow data in a preset format based on the statistical flow value and the statistical parameter value comprises:
setting the statistical parameter value as a Key value of the first flow data;
and setting the statistical flow Value as the Value of the first flow data.
4. The method of claim 2, wherein the querying the traffic database based on the at least one statistical parameter value to obtain second traffic data comprises:
converting the statistical parameter values into corresponding database query conditions;
splicing the database query conditions corresponding to the at least one statistical parameter value into a database query statement;
and querying the flow database through the database query statement to acquire second flow data.
5. The method of claim 1, wherein the extracting and displaying the corresponding first traffic data from the caching middleware based on the query request comprises:
acquiring query parameters in the query request;
querying a Key value of the first traffic data based on the query parameter;
and displaying the statistical flow value in the first flow data under the condition that the Key value is consistent with the query parameter.
6. The method of claim 5, wherein the query parameter comprises a flow generation time, and wherein displaying the statistical flow value in the first flow data if the Key value is consistent with the query parameter comprises:
determining a difference between a query time and the traffic generation time;
and acquiring corresponding first flow data based on the difference value.
7. The method of claim 1, wherein the statistical task comprises at least one statistical parameter value, wherein the statistical parameter value comprises a statistical period, wherein performing statistics on the traffic data in the traffic database based on the statistical task comprises:
determining the execution time of the statistical task based on the statistical period;
and counting the flow data generated in the counting period in the execution time, and acquiring first flow data corresponding to the counting period.
8. The method of claim 1, wherein after storing the first traffic data in caching middleware, the method further comprises:
setting a corresponding expiration time for the first traffic data;
deleting the first streaming data in case that a storage time of the first streaming data in the caching middleware exceeds the expiration time.
9. A flow data statistical device is applied to a flow data management system, and is characterized in that the flow data management system comprises a flow database, a cache middleware and a task statistical engine, and the device comprises:
the receiving module is used for receiving a statistic task of the flow data;
the statistic module is used for carrying out statistics on the flow data in the flow database based on the statistic task to obtain first flow data;
a storing module, configured to store the first traffic data into the caching middleware;
and the extraction module is used for extracting and displaying corresponding first flow data from the cache middleware based on the query request.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the traffic data statistical method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210846343.4A CN115314423A (en) | 2022-07-19 | 2022-07-19 | Traffic data statistical method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210846343.4A CN115314423A (en) | 2022-07-19 | 2022-07-19 | Traffic data statistical method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115314423A true CN115314423A (en) | 2022-11-08 |
Family
ID=83856795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210846343.4A Withdrawn CN115314423A (en) | 2022-07-19 | 2022-07-19 | Traffic data statistical method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115314423A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608207A (en) * | 2015-12-25 | 2016-05-25 | 广州华多网络科技有限公司 | Data statistics system based on Redis database and statistics method of data statistics system |
| CN109815214A (en) * | 2018-12-29 | 2019-05-28 | 深圳云天励飞技术有限公司 | Data bank access method, system, device and storage medium |
| CN112637305A (en) * | 2020-12-16 | 2021-04-09 | 平安消费金融有限公司 | Data storage and query method, device, equipment and medium based on cache |
| CN113282626A (en) * | 2021-05-31 | 2021-08-20 | 平安国际智慧城市科技股份有限公司 | Redis-based data caching method and device, computer equipment and storage medium |
-
2022
- 2022-07-19 CN CN202210846343.4A patent/CN115314423A/en not_active Withdrawn
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608207A (en) * | 2015-12-25 | 2016-05-25 | 广州华多网络科技有限公司 | Data statistics system based on Redis database and statistics method of data statistics system |
| CN109815214A (en) * | 2018-12-29 | 2019-05-28 | 深圳云天励飞技术有限公司 | Data bank access method, system, device and storage medium |
| CN112637305A (en) * | 2020-12-16 | 2021-04-09 | 平安消费金融有限公司 | Data storage and query method, device, equipment and medium based on cache |
| CN113282626A (en) * | 2021-05-31 | 2021-08-20 | 平安国际智慧城市科技股份有限公司 | Redis-based data caching method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109271411B (en) | Report generation method, report generation device, computer equipment and storage medium | |
| US8346758B2 (en) | Method and system for transmitting a query in a wireless network | |
| CN102915374B (en) | A kind of method, Apparatus and system of resource access of controlling database | |
| US10148782B2 (en) | Message consumer orchestration framework | |
| CN106033324B (en) | Data storage method and device | |
| CN109815214B (en) | Database access method, system, device and storage medium | |
| CN108243346B (en) | Method for filtering bullet screen information, server and computer equipment | |
| US8438275B1 (en) | Formatting data for efficient communication over a network | |
| WO2014173151A1 (en) | Method, device and terminal for data processing | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN110858192A (en) | Log query method and system, log checking system and query terminal | |
| WO2021244473A1 (en) | Frequency control method and apparatus | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN109446225B (en) | Data caching method and device, computer equipment and storage medium | |
| CN103530335A (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN111078651A (en) | Method and device for counting usage amount of object storage | |
| CN107943912B (en) | A kind of response type Resource TOC data visualization management method, terminal and device | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN104168174A (en) | Method and apparatus for information transmission | |
| CN115314423A (en) | Traffic data statistical method, device and storage medium | |
| CN105872731A (en) | Data processing method and device | |
| CN107846328B (en) | Network rate real-time statistical method based on concurrent lock-free ring queue | |
| CN106886563B (en) | Advertisement filtering method and device | |
| CN115525603A (en) | Storage statistics method and device, computer readable storage medium and AI device | |
| CN102377580A (en) | Method and equipment for uploading performance data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20221108 |