CN113407363A - Sliding window counting method and device based on remote dictionary service - Google Patents
Sliding window counting method and device based on remote dictionary service Download PDFInfo
- Publication number
- CN113407363A CN113407363A CN202110700095.8A CN202110700095A CN113407363A CN 113407363 A CN113407363 A CN 113407363A CN 202110700095 A CN202110700095 A CN 202110700095A CN 113407363 A CN113407363 A CN 113407363A
- Authority
- CN
- China
- Prior art keywords
- counting
- remote dictionary
- dictionary service
- target field
- sliding window
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/544—Remote
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The invention provides a sliding window counting method and a device based on remote dictionary service, wherein the method comprises the following steps: acquiring configuration information in a configuration service; receiving a counting message sent by a target program according to the configuration information; generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and a preset count message, and storing the target field and the count value in a remote dictionary service; and according to the sliding window counting request, counting and/or inquiring the target field and the corresponding counting value to obtain a target counting result. The method and the device for the sliding window counting have the advantages that the configuration information of the configuration service is obtained, so that the flexibility of configuration processing is high, the counting message is received according to the configuration information, the target field and the counting value are generated and stored based on the preset remote dictionary service, the technical frame and the component are not required to be used jointly, the logic is simple, the stability is high, the target field and the corresponding counting value are counted and/or inquired according to the sliding window counting request, and the sliding window counting with high flexibility of sliding window operation and configuration processing is realized.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a sliding window counting method and device based on remote dictionary service.
Background
The sliding window counting technology provides a simple and feasible solution for some scenes which need to carry out periodical statistics according to time movement and carry out real-time query on counting results, such as monitoring in a certain period, statistical control requirements and the like. In the prior art, a stream-introduced computing framework is mostly adopted to realize sliding window counting, which is generally used for solving the requirement of offline service function, and if the method is used in a real-time service process, a cache technology component is required to be added, and the cached data operation is required to be maintained.
The prior art needs to combine multiple technology frameworks and technology components, and has complex realization and low flexibility of sliding window operation and configuration processing.
Disclosure of Invention
The invention provides a sliding window counting method and device based on remote dictionary service, which are used for overcoming the defects of complexity and low flexibility of realization by using a plurality of technical frames and technical components in a combined manner in the prior art, and realizing the sliding window counting method with simple logic, strong stability and high flexibility of sliding window operation and configuration processing.
The invention provides a sliding window counting method based on remote dictionary service, which comprises the following steps:
acquiring configuration information in a configuration service;
receiving a counting message sent by a target program according to the configuration information;
generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and the count message, and storing the target field and the count value in the remote dictionary service;
and counting and/or inquiring the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
According to the sliding window counting method based on the remote dictionary service, provided by the invention, the configuration information comprises statistical dimensionality;
the receiving, according to the configuration information, a counting message sent by a target program includes:
and receiving counting information which is sent by the target program and corresponds to the counting dimension according to the counting dimension of the configuration information.
According to the sliding window counting method based on the remote dictionary service, provided by the invention, the configuration information comprises time granularity, and the counting message comprises a statistical dimension identifier, a counting unique identifier and a counting event time;
generating a corresponding target field based on a preset remote dictionary service storage rule and the counting message, including:
converting the counting event time in the counting message into a time character string in a corresponding format according to the time granularity of the configuration information;
and generating the statistical dimension identification, the counting unique identification and the time character string into a target field based on a preset remote dictionary service storage rule.
According to the sliding window counting method based on the remote dictionary service, the method for generating the corresponding count value based on the preset remote dictionary service storage rule and the counting message comprises the following steps:
if the target field does not exist in the remote dictionary service, initializing a count value corresponding to the target field to 1;
and if the target field exists in the remote dictionary service, increasing the count value corresponding to the target field by 1.
According to the sliding window counting method based on the remote dictionary service, provided by the invention, the target field and the corresponding counting value are counted and/or inquired according to the sliding window counting request to obtain a target counting result, and the method comprises the following steps:
and generating a query instruction of the remote dictionary service according to the sliding window counting request, and counting and/or querying the target field and the corresponding count value in the remote dictionary service based on the query instruction to obtain a target counting result.
According to the sliding window counting method based on the remote dictionary service, after the corresponding target field and the counting value are generated and stored in the remote dictionary service, the method further comprises the following steps:
and setting first expiration time for the target field, and deleting the target field and the corresponding count value in the remote dictionary service when the expiration time is reached.
According to the sliding window counting method based on the remote dictionary service, after the corresponding target field and the counting value are generated and stored in the remote dictionary service, the method further comprises the following steps:
converting the target field based on a preset conversion rule to generate a secondary field;
setting a second failure time for the secondary field based on the first failure time, wherein the second failure time is smaller than the first failure time;
receiving a failure event notification sent by a remote dictionary service based on the second failure time, wherein the failure event notification comprises a failure secondary field;
analyzing the failure secondary field to obtain a corresponding target field to be failed;
determining a count value corresponding to the target field to be invalidated, which is stored in a remote dictionary service, based on the target field to be invalidated;
and storing the target field to be invalidated and the corresponding count value to data storage equipment.
According to the sliding window counting method based on the remote dictionary service, after the target field to be invalidated and the corresponding count value are stored in the data storage device, the method further comprises the following steps:
and deleting the target field to be invalidated and the corresponding count value from the remote dictionary service.
The invention also provides a sliding window counting device based on remote dictionary service, which comprises:
the device comprises an acquisition configuration information unit, a configuration service unit and a configuration information unit, wherein the acquisition configuration information unit is used for acquiring configuration information in a configuration service;
a counting message receiving unit, configured to receive a counting message sent by a target program according to the configuration information;
the counting storage unit is used for generating a corresponding target field and a counting value based on a preset remote dictionary service storage rule and the counting message and storing the corresponding target field and the counting value in the remote dictionary service;
and the counting query unit is used for counting and/or querying the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
According to the sliding window counting device based on the remote dictionary service, the configuration information comprises statistical dimensionality;
the receive count message unit is further to:
and receiving counting information which is sent by the target program and corresponds to the counting dimension according to the counting dimension of the configuration information.
According to the sliding window counting device based on the remote dictionary service, the configuration information comprises time granularity, and the counting message comprises a statistic dimension identifier, a counting unique identifier and a counting event time;
the count storage unit is further configured to:
converting the counting event time in the counting message into a time character string in a corresponding format according to the time granularity of the configuration information;
and generating the statistical dimension identification, the counting unique identification and the time character string into a target field based on a preset remote dictionary service storage rule.
According to the sliding window counting device based on the remote dictionary service, provided by the invention, the counting storage unit is further used for:
if the target field does not exist in the remote dictionary service, initializing a count value corresponding to the target field to 1;
and if the target field exists in the remote dictionary service, increasing the count value corresponding to the target field by 1.
According to the sliding window counting device based on the remote dictionary service, provided by the invention, the statistical query unit is further configured to:
and generating a query instruction of the remote dictionary service according to the sliding window counting request, and counting and/or querying the target field and the corresponding count value in the remote dictionary service based on the query instruction to obtain a target counting result.
According to the sliding window counting device based on the remote dictionary service, the device further comprises a failure cleaning unit, wherein the failure cleaning unit is used for generating corresponding target fields and counting values and storing the counting values in the remote dictionary service:
and setting first expiration time for the target field, and deleting the target field and the corresponding count value in the remote dictionary service when the expiration time is reached.
According to the sliding window counting device based on the remote dictionary service, the device further comprises a historical data storage unit, wherein the historical data storage unit is used for generating corresponding target fields and counting values and storing the counting values in the remote dictionary service:
converting the target field based on a preset conversion rule to generate a secondary field;
setting a second failure time for the secondary field based on the first failure time, wherein the second failure time is smaller than the first failure time;
receiving a failure event notification sent by a remote dictionary service based on the second failure time, wherein the failure event notification comprises a failure secondary field;
analyzing the failure secondary field to obtain a corresponding target field to be failed;
determining a count value corresponding to the target field to be invalidated, which is stored in a remote dictionary service, based on the target field to be invalidated;
and storing the target field to be invalidated and the corresponding count value to data storage equipment.
According to the sliding window counting device based on the remote dictionary service, provided by the invention, the historical data storage unit is further used for:
and after the target field to be invalidated and the corresponding count value are stored in a data storage device, deleting the target field to be invalidated and the corresponding count value from the remote dictionary service.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the sliding window counting method based on the remote dictionary service.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the remote dictionary service based sliding window counting method as described in any one of the above.
The sliding window counting method and device based on the remote dictionary service, provided by the invention, have the advantages that the configuration information of the configuration service is obtained, so that the flexibility of the configuration processing is high, the counting message sent by a target program is received according to the configuration information, then the corresponding target field and the counting value are generated and stored in the remote dictionary service based on the preset storage rule and the counting message of the remote dictionary service, various technical frames and technical components are not required to be used in a combined manner, the logic is simple, the stability is high, and then the counting and/or query are carried out on the target field and the corresponding counting value according to the sliding window counting request, so that the sliding window counting with high flexibility of the sliding window operation and the configuration processing is realized.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flowchart illustrating a sliding window counting method based on a remote dictionary service according to an embodiment of the present invention;
FIG. 2 is a second flowchart illustrating a sliding window counting method based on remote dictionary services according to an embodiment of the present invention;
FIG. 3 is a third flowchart illustrating a sliding window counting method based on remote dictionary services according to an embodiment of the present invention;
FIG. 4 is a fourth flowchart illustrating a sliding window counting method based on remote dictionary services according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the interaction between the services provided by the embodiment of the present invention;
FIG. 6 is a schematic structural diagram of a sliding window counting apparatus based on a remote dictionary service according to the present invention;
fig. 7 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The technical solutions provided by the embodiments of the present invention are described below with reference to fig. 1 to 7.
Fig. 1 is a schematic flowchart of a sliding window counting method based on a remote dictionary service according to an embodiment of the present invention, where the method includes:
And step 120, receiving a counting message sent by the target program according to the configuration information.
Specifically, the computing server cluster periodically pulls configuration information related to the count from the configuration service and caches the configuration information to the local memory. And the application program or other data acquisition programs carry out counting and point burying, and when the counting mechanism is triggered, the application program or other data acquisition programs send counting messages to the counting server cluster. And the computing server cluster receives the counting message sent by the target program according to the configuration information.
And step 130, generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and a preset count message, and storing the target field and the count value in the remote dictionary service.
Among them, Remote Dictionary service (Remote Dictionary Server, Redis) is an open-source log-type, Key-Value database written in ANSI C language, supporting network, and based on memory and can be persistent. And processing the counting message to obtain a target field in a Key-Value format and a count Value, and storing the target field and the count Value in Redis to realize the quick query of the counting data.
And 140, counting and/or inquiring the target field and the corresponding count value according to the sliding window counting request to obtain a target counting result.
Specifically, the computing server cluster provides an API (application programming interface) for an external application program, provides counting results required by the external application program for inquiry, can support time point inquiry and counting statistics in a certain time period, and simultaneously realizes the effect of sliding window counting so as to meet different counting use scenes of an application layer.
The sliding window counting method based on the remote dictionary service, provided by the invention, has the advantages that the configuration information in the configuration service is obtained, so that the configuration processing flexibility is high, the counting message sent by a target program is received according to the configuration information, then the corresponding target field and the counting value are generated and stored in the remote dictionary service based on the preset remote dictionary service storage rule and the counting message, various technical frames and technical components are not required to be used in a combined manner, the logic is simple, the stability is strong, and then the target field and the corresponding counting value are counted and/or inquired according to the sliding window counting request, so that the sliding window counting with high flexibility in sliding window operation and configuration processing is realized.
Further, in an embodiment of the present invention, the configuration information includes a statistical dimension and a time granularity.
Specifically, the newly added counting statistic dimension is configured through a configuration service, including selecting and configuring the time granularity of counting, and setting a statistic dimension identifier KSTR. And the computing server cluster receives counting information of the corresponding statistical dimension sent by the target program according to the statistical dimension of the configuration information.
Further, in an embodiment of the present invention, the counting message includes a statistics dimension identifier KSTR, a count unique identifier CUSTR, and a count event time DateTime.
Correspondingly, step 130 specifically includes:
Specifically, the DateTime is obtained from the counting message, if the counting message is a null value, the current time of the server is taken as the DateTime, the configuration information is obtained based on the KSTR in the counting message to determine the time granularity of the corresponding KSTR, and the DateTime is converted into a time string in a different format based on the time granularity, such as seconds: YYYYMMDDHHMMSS, min: YYYYMMDDHHMM, hour: YYYYMMDDHH are provided.
And step 220, generating a target field from the statistical dimension identifier, the counting unique identifier and the time character string based on a preset remote dictionary service storage rule.
Specifically, the KSTR and the time string are combined to form the target field Key of the count data, for example, Key ═ KSTR + "@" + CUSTR + "@" + time string.
For the count value corresponding to the target field, whether the current target field exists in Redis needs to be judged, and if the current target field does not exist, the count value corresponding to the target field is increased by 1; and if so, initializing the count value corresponding to the target field to 1.
It should be noted that the above storage rule implements the cumulative counting of counting messages within the same time granularity, and generates the effect of "sliding count" as time goes by.
Further, in an embodiment of the present invention, the step 140 specifically includes:
and generating a query instruction of Redis according to the sliding window counting request, and counting and/or querying the target field and the corresponding counting value in the Redis based on the query instruction to obtain a target counting result.
Specifically, the computing server cluster provides an API interface to obtain a query request of an external application program. For time point query of time granularity, Redis uses get instruction to process; for counting statistics in a certain time period, such as 5 seconds, 20 hours, etc., Redis uses the mget instruction to query, and for a super-large time period of 200 hours, in order to improve the performance of the statistics, the configured counting time granularity should be upgraded to perform the statistics.
It should be noted that, based on a preset Redis storage rule and a counting message, a corresponding target field and a corresponding count value are generated and stored in the Redis, and a sliding window counting capability can be formed by performing statistical processing on a counting period in a query process.
Further, in an embodiment of the present invention, after generating and storing the corresponding target field and the count value in Redis, the method further includes:
and setting a first expiration time for the target field, and deleting the target field and the corresponding count value in Redis when the expiration time is reached.
Specifically, the target field is set with an expired automatic cleaning time which can be randomly increased by an increment above a fixed evaluation expiration time of a time period granularity, for example, the time period of "second" is 60 seconds, and then an increment of 10-30 seconds is added to avoid that a large number of target fields occur at the same time to trigger an aging mechanism.
Further, in an embodiment of the present invention, in order to increase the use of the history data, the past due expiration history may be stored, which specifically includes:
and 310, converting the target field based on a preset conversion rule to generate a secondary field.
And 320, setting second failure time for the secondary field based on the first failure time, wherein the second failure time is less than the first failure time.
And step 330, receiving an invalidation event notification sent by Redis based on the second invalidation time, wherein the invalidation event notification comprises an invalidation secondary field.
And 340, analyzing the failure secondary field to obtain a corresponding target field to be failed.
And step 350, determining a count value corresponding to the target field to be invalidated, which is stored in Redis, based on the target field to be invalidated.
And step 360, storing the target field to be invalidated and the corresponding count value to the data storage device.
Specifically, a secondary field EKey is defined, which may be in the form of EKey ═ E + "#" + Key, and the expiration time of the EKey is set to be above the fixed evaluation time of the time period granularity, and is randomly increased by 100-1000 ms, where it is to be noted that the random increment may be another value smaller than the first expiration time. And then, the computing server cluster monitors an EKey failure event notification of Redis, and filters out a failure secondary field EKey which needs to be processed, because the notification event can only acquire a field but cannot acquire a corresponding count value, a Key needs to be analyzed according to the EKey, and then the Key is acquired from a cache and written into a data storage device for subsequent query.
It should be noted that after the target field to be invalidated and the corresponding count value are stored in the data storage device, the target field to be invalidated in the Redis may be actively deleted, or the corresponding target field to be invalidated and the count value in the Redis may be cleared when the first invalidation time is reached.
In summary, the sliding window counting method based on remote dictionary service provided by the present invention obtains the configuration information in the configuration service, thereby, the flexibility of the configuration processing is high, and according to the configuration information, the counting message sent by the target program is received, then based on the preset remote dictionary service storage rule and the counting message, generating a corresponding target field and a counting value and storing the corresponding target field and the counting value in the remote dictionary service, the accumulated counting of the counting messages in the same time granularity is realized, a plurality of technical frameworks and technical components are not required to be used jointly, the logic is simple, the stability is strong, and then according to the sliding window counting request, and counting the counting period of the target field and the corresponding counting value, and providing automatic cleaning of expired failure data, thereby realizing sliding window counting with high flexibility in sliding window operation and configuration processing.
An embodiment of the present invention further provides a sliding window counting method based on a remote dictionary service, referring to fig. 4 and 5, the method includes:
Specifically, as shown in fig. 5, the configuration service provides a configured function management interface, so that the user can configure the newly added counting statistical dimension, including selecting and configuring the time granularity of counting, such as seconds, minutes, and hours, and setting the statistical dimension identifier KSTR.
And step 420, receiving a counting message of the corresponding statistical dimension sent by the target program according to the statistical dimension of the configuration information.
Specifically, as shown in fig. 5, the compute server cluster periodically pulls configuration information related to the count from the configuration service and caches the configuration information in the local memory. And counting and embedding the points by the application program or other data acquisition programs, and when the counting mechanism is triggered, generating a message by the application program or other data acquisition programs, putting the message into a message queue, and waiting for sending the message to the counting server cluster. And the computing server cluster receives counting information of the corresponding statistical dimension sent by the target program according to the statistical dimension of the configuration information.
Specifically, in the computation count processing logic of fig. 5, the computation server obtains DateTime from the count message, if the count message is null, then takes the current time of the server as DateTime, obtains configuration information based on the KSTR in the count message to determine the time granularity of the corresponding KSTR, and converts the DateTime into a time string in a different format, such as seconds, based on the time granularity: YYYYMMDDHHMMSS, min: YYYYMMDDHHMM, hour: YYYYMMDDHH are provided.
And step 440, generating a target field from the statistical dimension identifier, the counting unique identifier and the time character string based on a preset Redis storage rule, and determining a count value corresponding to the target field.
Specifically, the Key of the count data is formed by combining the KSTR and the time string, for example, the Key ═ KSTR + "@" + CUSTR + "@ + time string.
For the count value corresponding to the target field, whether the current target field exists in Redis needs to be judged, and if the current target field does not exist, the count value corresponding to the target field is increased by 1; and if so, initializing the count value corresponding to the target field to 1.
And step 450, setting first expiration time for the target field, and deleting the target field and the corresponding count value in Redis when the expiration time is reached.
Specifically, the target field is set with an expired automatic cleaning time which can be randomly increased by an increment above a fixed evaluation expiration time of a time period granularity, for example, the time period of "second" is 60 seconds, and then an increment of 10-30 seconds is added to avoid that a large number of target fields occur at the same time to trigger an aging mechanism.
step 461, converting the target field based on a preset conversion rule to generate a secondary field.
Step 462, set a second expiration time for the secondary field based on the first expiration time, the second expiration time being less than the first expiration time.
And step 463, receiving a failure event notification sent by Redis based on the second failure time, wherein the failure event notification comprises a failure secondary field.
And 464, analyzing the failure secondary field to obtain a corresponding target field to be failed.
And 465, determining the count value of the corresponding to-be-invalidated target field stored in Redis based on the to-be-invalidated target field.
Step 466, storing the target field to be invalidated and the corresponding count value to the data storage device, and deleting the target field to be invalidated from the Redis.
Specifically, a secondary field EKey is defined, which is in the form of EKey ═ E + "#" + Key, the expiration time of the EKey is set to be above the fixed evaluation time of the time period granularity, and the expiration time is randomly increased by 100-1000 ms, it should be noted that the random increment may be other values smaller than the first expiration time. Then, as shown in fig. 5, the compute server cluster monitors Ekey failure event notification of Redis, and filters an Ekey that needs to be processed, because the notification event can only obtain a field but cannot obtain a corresponding count value, a Key needs to be parsed according to the Ekey, and then a corresponding count value is obtained from a cache and written into the data storage device for subsequent query.
Specifically, as shown in fig. 5, the compute server cluster provides an API interface to obtain a query request of an external application and returns a query result. For time point query of time granularity, Redis uses get instruction to process; for counting statistics in a certain time period, such as 5 seconds, 20 hours, etc., Redis uses the mget instruction to perform query, and for an oversized time period of 200 hours, in order to improve the performance of the statistics, the configured counting time granularity should be upgraded. And the calculation server receives the time point counting result or the counting results of a plurality of time points in a certain time period to perform time period counting calculation, and returns the counting result to the external application program through a calculation service API.
In summary, the sliding window counting method based on the remote dictionary service provided by the invention obtains the configuration information in the configuration service, so that the flexibility of configuration processing is high, the counting message sent by the target program is received according to the configuration information, then the corresponding target field and the counting value are generated and stored in the remote dictionary service based on the preset remote dictionary service storage rule and the counting message, the accumulated counting of the counting message in the same time granularity is realized, various technical frames and technical components are not required to be used jointly, the logic is simple, the stability is strong, then the counting period of the target field and the corresponding counting value is counted according to the sliding window counting request, the automatic clearing of expired failure data is provided, and the sliding window counting with high flexibility of sliding window operation and configuration processing is realized.
The following describes the sliding window counting device based on the remote dictionary service provided by the present invention, and the sliding window counting device based on the remote dictionary service described below and the sliding window counting method based on the remote dictionary service described above can be referred to correspondingly.
Fig. 6 is a schematic structural diagram of a sliding window counting apparatus based on a remote dictionary service according to an embodiment of the present invention, as shown in fig. 6, the apparatus includes:
an acquire configuration information unit 610, configured to acquire configuration information in a configuration service;
a receive count message unit 620, configured to receive a count message sent by the target program according to the configuration information;
a count storage unit 630, configured to generate a corresponding target field and a count value based on a preset remote dictionary service storage rule and the count message, and store the target field and the count value in the remote dictionary service;
and the statistics query unit 640 is configured to perform statistics and/or query on the target field and the corresponding count value according to the sliding window counting request, so as to obtain a target counting result.
The sliding window counting device based on the remote dictionary service, provided by the invention, acquires the configuration information in the configuration service, so that the flexibility of configuration processing is high, the counting information sent by a target program is received according to the configuration information, and then the corresponding target field and the counting value are generated and stored in the remote dictionary service based on the preset storage rule and the counting information of the remote dictionary service, multiple technical frames and technical components are not required to be used in a combined manner, the logic is simple, the stability is strong, and then the target field and the corresponding counting value are counted and/or inquired according to the sliding window counting request, so that the sliding window counting with high flexibility of sliding window operation and configuration processing is realized.
Further, in an embodiment of the present invention, the configuration information includes a statistical dimension;
the receive count message unit 620 is further configured to:
and receiving counting information which is sent by the target program and corresponds to the counting dimension according to the counting dimension of the configuration information.
Further, in an embodiment of the present invention, the configuration information includes time granularity, and the counting message includes a statistical dimension identifier, a unique counting identifier, and a counting event time;
the count storage unit 630 is further configured to:
converting the counting event time in the counting message into a time character string in a corresponding format according to the time granularity of the configuration information;
and generating the statistical dimension identification, the counting unique identification and the time character string into a target field based on a preset remote dictionary service storage rule.
Further, in an embodiment of the present invention, the count storage unit 630 is further configured to:
if the target field does not exist in the remote dictionary service, initializing a count value corresponding to the target field to 1;
and if the target field exists in the remote dictionary service, increasing the count value corresponding to the target field by 1.
Further, in an embodiment of the present invention, the statistical query unit 640 is further configured to:
and generating a query instruction of the remote dictionary service according to the sliding window counting request, and counting and/or querying the target field and the corresponding count value in the remote dictionary service based on the query instruction to obtain a target counting result.
Further, in an embodiment of the present invention, the apparatus further includes a failure clearing unit, configured to, after generating and storing a corresponding target field and a count value in the remote dictionary service:
and setting first expiration time for the target field, and deleting the target field and the corresponding count value in the remote dictionary service when the expiration time is reached.
Further, in an embodiment of the present invention, the apparatus further includes a historical data storage unit, configured to, after generating and storing a corresponding target field and a count value in the remote dictionary service:
converting the target field based on a preset conversion rule to generate a secondary field;
setting a second failure time for the secondary field based on the first failure time, wherein the second failure time is smaller than the first failure time;
receiving a failure event notification sent by a remote dictionary service based on the second failure time, wherein the failure event notification comprises a failure secondary field;
analyzing the failure secondary field to obtain a corresponding target field to be failed;
determining a count value corresponding to the target field to be invalidated, which is stored in a remote dictionary service, based on the target field to be invalidated;
and storing the target field to be invalidated and the corresponding count value to data storage equipment.
Further, in an embodiment of the present invention, the history data storage unit is further configured to:
and after the target field to be invalidated and the corresponding count value are stored in a data storage device, deleting the target field to be invalidated and the corresponding count value from the remote dictionary service.
In summary, the sliding window counting apparatus based on the remote dictionary service provided by the present invention obtains the configuration information in the configuration service, so that the flexibility of the configuration processing is high, receives the counting message sent by the target program according to the configuration information, generates the corresponding target field and the counting value based on the preset storage rule of the remote dictionary service and the counting message, and stores the corresponding target field and the counting value in the remote dictionary service, thereby realizing the accumulated counting of the counting message in the same time granularity, without using multiple technical frames and technical components in combination, the logic is simple, the stability is strong, and then according to the sliding window counting request, the counting period of the target field and the corresponding counting value is counted, and simultaneously, the automatic cleaning of the expired failure data is provided, thereby realizing the sliding window counting with high flexibility of the sliding window operation and the configuration processing.
Fig. 7 illustrates a physical structure diagram of an electronic device, and as shown in fig. 7, the electronic device may include: a processor (processor)710, a communication Interface (Communications Interface)720, a memory (memory)830 and a communication bus 740, wherein the processor 710, the communication Interface 720 and the memory 730 communicate with each other via the communication bus 740. Processor 710 may invoke logic instructions in memory 730 to perform a remote dictionary service based sliding window counting method comprising: acquiring configuration information in a configuration service; receiving a counting message sent by a target program according to the configuration information; generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and the count message, and storing the target field and the count value in the remote dictionary service; and counting and/or inquiring the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
In addition, the logic instructions in the memory 730 can be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer being capable of executing the sliding window counting method based on a remote dictionary service provided in the above embodiments, the method including: acquiring configuration information in a configuration service; receiving a counting message sent by a target program according to the configuration information; generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and the count message, and storing the target field and the count value in the remote dictionary service; and counting and/or inquiring the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the method for remote dictionary service-based sliding window counting provided in the above embodiments, the method comprising: acquiring configuration information in a configuration service; receiving a counting message sent by a target program according to the configuration information; generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and the count message, and storing the target field and the count value in the remote dictionary service; and counting and/or inquiring the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (11)
1. A sliding window counting method based on remote dictionary service is characterized by comprising the following steps:
acquiring configuration information in a configuration service;
receiving a counting message sent by a target program according to the configuration information;
generating a corresponding target field and a corresponding count value based on a preset remote dictionary service storage rule and the count message, and storing the target field and the count value in the remote dictionary service;
and counting and/or inquiring the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
2. The remote dictionary service-based sliding window counting method according to claim 1, wherein the configuration information includes statistical dimensions;
the receiving, according to the configuration information, a counting message sent by a target program includes:
and receiving counting information which is sent by the target program and corresponds to the counting dimension according to the counting dimension of the configuration information.
3. The remote dictionary service-based sliding window counting method according to claim 1, wherein the configuration information includes time granularity, the counting message includes a statistical dimension identification, a counting unique identification and a counting event time;
generating a corresponding target field based on a preset remote dictionary service storage rule and the counting message, including:
converting the counting event time in the counting message into a time character string in a corresponding format according to the time granularity of the configuration information;
and generating the statistical dimension identification, the counting unique identification and the time character string into a target field based on a preset remote dictionary service storage rule.
4. The sliding window counting method based on the remote dictionary service according to claim 1, wherein the generating a corresponding count value based on the preset remote dictionary service storage rule and the counting message comprises:
if the target field does not exist in the remote dictionary service, initializing a count value corresponding to the target field to 1;
and if the target field exists in the remote dictionary service, increasing the count value corresponding to the target field by 1.
5. The sliding window counting method based on remote dictionary service according to claim 1, wherein the counting and/or querying the target field and the corresponding count value according to the sliding window counting request to obtain the target counting result comprises:
and generating a query instruction of the remote dictionary service according to the sliding window counting request, and counting and/or querying the target field and the corresponding count value in the remote dictionary service based on the query instruction to obtain a target counting result.
6. The remote dictionary service-based sliding window counting method of claim 1, wherein after generating and storing corresponding target fields and count values in the remote dictionary service, the method further comprises:
and setting first expiration time for the target field, and deleting the target field and the corresponding count value in the remote dictionary service when the expiration time is reached.
7. The remote dictionary service-based sliding window counting method of claim 6, wherein after generating and storing corresponding target fields and count values in the remote dictionary service, the method further comprises:
converting the target field based on a preset conversion rule to generate a secondary field;
setting a second failure time for the secondary field based on the first failure time, wherein the second failure time is smaller than the first failure time;
receiving a failure event notification sent by a remote dictionary service based on the second failure time, wherein the failure event notification comprises a failure secondary field;
analyzing the failure secondary field to obtain a corresponding target field to be failed;
determining a count value corresponding to the target field to be invalidated, which is stored in a remote dictionary service, based on the target field to be invalidated;
and storing the target field to be invalidated and the corresponding count value to data storage equipment.
8. The remote dictionary service-based sliding window counting method according to claim 7, wherein after storing the target-to-be-invalidated field and the corresponding count value in a data storage device, the method further comprises:
and deleting the target field to be invalidated and the corresponding count value from the remote dictionary service.
9. A sliding window counting device based on remote dictionary service is characterized by comprising:
the device comprises an acquisition configuration information unit, a configuration service unit and a configuration information unit, wherein the acquisition configuration information unit is used for acquiring configuration information in a configuration service;
a counting message receiving unit, configured to receive a counting message sent by a target program according to the configuration information;
the counting storage unit is used for generating a corresponding target field and a counting value based on a preset remote dictionary service storage rule and the counting message and storing the corresponding target field and the counting value in the remote dictionary service;
and the counting query unit is used for counting and/or querying the target field and the corresponding counting value according to the sliding window counting request to obtain a target counting result.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program performs the steps of the remote dictionary service based sliding window counting method according to any one of claims 1 to 8.
11. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, performs the steps of the remote dictionary service based sliding window counting method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110700095.8A CN113407363A (en) | 2021-06-23 | 2021-06-23 | Sliding window counting method and device based on remote dictionary service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110700095.8A CN113407363A (en) | 2021-06-23 | 2021-06-23 | Sliding window counting method and device based on remote dictionary service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113407363A true CN113407363A (en) | 2021-09-17 |
Family
ID=77682708
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110700095.8A Pending CN113407363A (en) | 2021-06-23 | 2021-06-23 | Sliding window counting method and device based on remote dictionary service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113407363A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114745566A (en) * | 2022-04-15 | 2022-07-12 | 北京搜狐新动力信息技术有限公司 | Management method of teenager mode, related device and computer storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090089048A1 (en) * | 2007-09-28 | 2009-04-02 | Microsoft Corporation | Two-Pass Hash Extraction of Text Strings |
| US20120143894A1 (en) * | 2010-12-02 | 2012-06-07 | Microsoft Corporation | Acquisition of Item Counts from Hosted Web Services |
| CN104866597A (en) * | 2015-05-29 | 2015-08-26 | 北京奇虎科技有限公司 | Method and apparatus for analyzing database access request |
| WO2019001074A1 (en) * | 2017-06-30 | 2019-01-03 | 武汉斗鱼网络科技有限公司 | Remote process calling method and apparatus, and computer device |
| US20190089370A1 (en) * | 2016-12-13 | 2019-03-21 | Hefei University Of Technology | Program counter compression method and hardware circuit thereof |
| CN110637341A (en) * | 2017-03-24 | 2019-12-31 | 加尔文医学研究所 | Processing of sequence data streams |
| CN112069773A (en) * | 2020-07-23 | 2020-12-11 | 北京三快在线科技有限公司 | Data processing system, method, apparatus, electronic device, and computer-readable medium |
| CN112688837A (en) * | 2021-03-17 | 2021-04-20 | 中国人民解放军国防科技大学 | Network measurement method and device based on time sliding window |
-
2021
- 2021-06-23 CN CN202110700095.8A patent/CN113407363A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090089048A1 (en) * | 2007-09-28 | 2009-04-02 | Microsoft Corporation | Two-Pass Hash Extraction of Text Strings |
| US20120143894A1 (en) * | 2010-12-02 | 2012-06-07 | Microsoft Corporation | Acquisition of Item Counts from Hosted Web Services |
| CN104866597A (en) * | 2015-05-29 | 2015-08-26 | 北京奇虎科技有限公司 | Method and apparatus for analyzing database access request |
| US20190089370A1 (en) * | 2016-12-13 | 2019-03-21 | Hefei University Of Technology | Program counter compression method and hardware circuit thereof |
| CN110637341A (en) * | 2017-03-24 | 2019-12-31 | 加尔文医学研究所 | Processing of sequence data streams |
| WO2019001074A1 (en) * | 2017-06-30 | 2019-01-03 | 武汉斗鱼网络科技有限公司 | Remote process calling method and apparatus, and computer device |
| CN112069773A (en) * | 2020-07-23 | 2020-12-11 | 北京三快在线科技有限公司 | Data processing system, method, apparatus, electronic device, and computer-readable medium |
| CN112688837A (en) * | 2021-03-17 | 2021-04-20 | 中国人民解放军国防科技大学 | Network measurement method and device based on time sliding window |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114745566A (en) * | 2022-04-15 | 2022-07-12 | 北京搜狐新动力信息技术有限公司 | Management method of teenager mode, related device and computer storage medium |
| CN114745566B (en) * | 2022-04-15 | 2023-10-20 | 北京搜狐新动力信息技术有限公司 | Teenager mode management method, related device and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111124819B (en) | Method and device for full link monitoring | |
| EP2383652A1 (en) | Performance monitoring for virtual machines | |
| US10389801B2 (en) | Service request processing method, related apparatus, and system | |
| CN109905286B (en) | Method and system for monitoring running state of equipment | |
| US20130204650A1 (en) | System and method for compliance management | |
| CN108415811B (en) | Method and device for monitoring business logic | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN110519263B (en) | Anti-swipe method, device, apparatus, and computer-readable storage medium | |
| CN110620699A (en) | Message arrival rate determination method, device, equipment and computer readable storage medium | |
| CN114090366A (en) | Method, device and system for monitoring data | |
| CN113407363A (en) | Sliding window counting method and device based on remote dictionary service | |
| US11709619B2 (en) | Method, electronic device, and computer program product for processing data | |
| CN114091704B (en) | Alarm suppression method and device | |
| CN109901918B (en) | Method and device for processing overtime task | |
| CN112019622A (en) | Flow control method, system, equipment and readable storage medium | |
| CN111131382A (en) | Message monitoring method and device | |
| CN113645260A (en) | Service retry method, device, storage medium and electronic equipment | |
| CN111291367A (en) | Access control method and system, electronic device and storage medium | |
| CN111611131B (en) | Operation and maintenance method, device and system based on Saltstack and storage medium | |
| CN113014675A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112463864A (en) | Data processing method and device and data processing system | |
| CN114090397A (en) | Alarm information processing method and device | |
| CN113329010B (en) | User access management method and system | |
| CN113535768A (en) | Production monitoring method and device | |
| CN113590437B (en) | Alarm information processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |