CN117834204A - Independent user number calculation method and device under continuous time constraint - Google Patents
Independent user number calculation method and device under continuous time constraint Download PDFInfo
- Publication number
- CN117834204A CN117834204A CN202311713501.XA CN202311713501A CN117834204A CN 117834204 A CN117834204 A CN 117834204A CN 202311713501 A CN202311713501 A CN 202311713501A CN 117834204 A CN117834204 A CN 117834204A
- Authority
- CN
- China
- Prior art keywords
- data
- queue
- log
- reporting
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims description 8
- 238000000034 method Methods 0.000 claims abstract description 30
- 235000016936 Dendrocalamus strictus Nutrition 0.000 claims abstract description 5
- 238000004458 analytical method Methods 0.000 claims abstract description 5
- 238000004590 computer program Methods 0.000 claims description 11
- 230000002776 aggregation Effects 0.000 claims description 9
- 238000004220 aggregation Methods 0.000 claims description 9
- 230000000903 blocking effect Effects 0.000 claims description 7
- 238000007405 data analysis Methods 0.000 claims description 7
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 230000001960 triggered effect Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000012546 transfer Methods 0.000 claims description 4
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000007619 statistical method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 235000010627 Phaseolus vulgaris Nutrition 0.000 description 1
- 244000046052 Phaseolus vulgaris Species 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001680 brushing effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000002674 ointment Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Abstract
The invention relates to a method and a device for calculating the number of independent users under continuous time constraint, belonging to the technical field of big data, wherein the method comprises the following steps: the method comprises the steps that logs needing to be energized by the anti-hotlink are sent to a data processing center queue of a content delivery network CDN, the data processing center analyzes IP information of a client Internet protocol, records needing to be reported are sent to a reporting queue, a data reporting program reports the reporting queue to a console, the console analyzes the reported records, performs service analysis on a whitelist, and seals and bans IP information in a non-whitelist. The invention also provides an independent user quantity calculating device and a storage medium under continuous time constraint.
Description
Technical Field
The invention belongs to the field of big data processing, and particularly relates to a method and a device for calculating the number of independent users under continuous time constraint.
Background
In the context of big data, customer resources are sometimes accessed maliciously and traffic is stolen. In order to prevent resources from being accessed maliciously, traffic is stolen and brushed, the requirement of client timestamp anti-theft chain needs to be met, and the requirement needs to support a memory database of high QPS as a temporary storage middleware, so that high concurrency competition exists in data processing. In the prior art, how to prevent the customer resources from being accessed and the traffic from being stolen is a technical problem to be solved currently.
Disclosure of Invention
In view of the above shortcomings of the prior art, an object of the present invention is to provide a method and an apparatus for calculating the number of independent users under continuous time constraint, so as to prevent malicious access to resources of clients and theft and brushing of traffic.
The first aspect of the present invention provides a method for calculating the number of independent users under continuous time constraint, including:
collecting log data, and sending the log which needs to be subjected to anti-hotlink enabling to a data processing center queue of a content delivery network CDN;
the data processing center analyzes the IP information of the client Internet protocol and sends the record to be reported to a reporting queue;
the data reporting program reports the reporting queue to a control console;
and the control console analyzes the reported records, analyzes the service of the white list and seals and bans the IP information in the non-white list.
Further, collecting log data includes:
the log collector collects the edge log files, packages the log data into a hypertext transfer protocol (HTTP) request according to the sequence of the generation time, and sends the HTTP request to a load balancer of the configured edge log access cluster.
Further, sending the log that the hotlink enabling is required to be performed to the data processing center queue of the CDN includes:
and the shunting program of the processing center sends the log which needs to be energized by the anti-theft chain to a CDN data processing center queue, and if the sending fails, retries according to a preset strategy until the sending is successful.
Further, the data processing center analyzing the client internet protocol IP information includes:
splitting the uniform resource locator URL into a user authentication identifier and a client unique identifier;
counting the number of client IP and service time splices according to the split dimension;
the split latitude is calculated into a character string with 32 bits according to MD5, the character string is stored into a database Redis as a group of key word storage values, locking operation is carried out during storage, and when a lock is obtained, whether the same key word exists is inquired from the database;
if the keyword does not exist, carrying out data analysis and conversion on a value corresponding to the keyword, combining the aggregation latitude into the recorded keyword, splicing and assembling the client IP and the service time into an object set, sequencing and intercepting the set, and when the size of the set is larger than the preset length, intercepting only the preset length and storing the set into a database, otherwise, storing the whole set into the database;
if the keyword exists, a value corresponding to the keyword is taken out, the aggregation latitude is formed into the recorded keyword, the client IP and the service time are spliced and assembled into an object set, the object set and the set in the database are combined, sorting and intercepting are carried out, when the size of the set is larger than the preset length, only the preset length is intercepted and stored in the database, and otherwise, the whole set is stored in the database;
wherein the aggregate dimension comprises one or a combination of: IP addresses, ports, resource addresses,
user authentication information and a limited number of times.
Further, the method further comprises the following steps: and when the length of the white list set is greater than or equal to the preset length, sending the data to a message queue of a report library, and storing the data in a database, and if the length of the white list set is less than the preset length, storing the data in the database only.
Further, the data reporting program reporting the report queue to the console includes:
subscribing a report queue by a data report program, packaging records reported by a statistical analysis program into HTTP requests, and reporting to a console;
wherein the package send request is triggered when a lot size condition or an interval time condition is satisfied.
Further, the control console analyzes the reported record, analyzes the service of the white list, and seals the IP information in the non-white list, which comprises:
when the control console receives the report records packed by the report program, analyzing the report records, carrying out service analysis on a white list, and registering the white list for the gateway, wherein the blocking duration is determined by an expiration time stamp expireTs in the report records;
request refusals are given to records not in the whitelist.
In a second aspect of the present invention, there is provided an independent user number calculation apparatus under continuous time constraint, comprising:
the log module is configured to collect log data and send the log which needs to be subjected to anti-hotlink enabling to a data processing center queue of the content delivery network CDN;
the data processing center is configured to analyze the IP information of the client Internet protocol and send the record to be reported to the reporting queue;
the reporting module is configured to report the reporting queue to a control console;
and the control console is configured to analyze the reported records, analyze the service of the white list and register the white list of the gateway.
In a third aspect, embodiments of the present application further provide an apparatus for calculating the number of independent users under continuous time constraint, including: a memory, a processor, and a user interface;
the memory is used for storing a computer program;
the user interface is used for realizing interaction with a user;
the processor is used for reading the computer program in the memory, and when the processor executes the computer program, the method for calculating the number of independent users under the continuous time constraint is realized.
In a fourth aspect, embodiments of the present application further provide a processor readable storage medium, where the processor readable storage medium stores a computer program, where the processor executes the computer program to implement the method for calculating the number of independent users under continuous time constraint provided by the present invention.
The invention has the following beneficial effects:
according to the method and the device, the high-availability big data analysis component is used, so that the stability and the performance of data acquisition are guaranteed, the consumption capacity of the data is improved, and backlog of the data is prevented.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, like reference numerals being used to refer to like parts throughout the several views. It is apparent that the drawings in the following description are only some of the embodiments described in the embodiments of the present invention, and that other drawings may be obtained from these drawings by those of ordinary skill in the art.
FIG. 1 is a diagram illustrating malicious access to a resource in the prior art;
FIG. 2 is a schematic diagram of an independent user number calculation architecture under continuous time constraint according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a method for calculating the number of independent users under continuous time constraint according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an apparatus for calculating the number of independent users under continuous time constraint according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an apparatus for calculating the number of independent users under another continuous time constraint according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the embodiments of the present invention better understood by those skilled in the art, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, shall fall within the scope of the invention.
In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
In the description of the present invention, it should be noted that unless explicitly stated and limited otherwise, the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The terms "mounted," "connected," "coupled," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of methods and systems that are consistent with aspects of the invention as detailed in the accompanying claims.
1. In the embodiment of the invention, the term "and/or" describes the association relation of the association objects, which means that three relations can exist, for example, a and/or B can be expressed as follows: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
2. The term "plurality" in embodiments of the present invention means two or more, and other adjectives are similar.
3. The anti-theft chain is software which adopts server programming and realizes the anti-theft chain through a uniform resource location url filtering technology. The content is provided to the end user directly on the page of advertising benefit by technical means, bypassing the final page of advertising benefit, instead of on the server itself.
4. NIO: non-blocking I/O, i.e., non-blocking I/O.
5. CDN: content Delivery Network, content distribution network.
6. key: keywords or keys.
7. value: the value refers to a value corresponding to the key.
Redis is a key and value storage system. It supports relatively more value types stored, including string, list, set, zset and hash. These data types all support push/pop, add/remove, and pick intersection union and difference and richer operations, and these operations are all atomic. On this basis, redis supports a variety of different ways of ordering. Redis data is cached in memory. Redis will periodically write updated data to disk or modify operation to additional record file, and master-slave synchronization is implemented on this basis.
Redis supports master-slave synchronization. The data may be synchronized from the master server to any number of slave servers, which may be master servers associated with other slave servers. Redis may perform single-level tree replication. The inventory may write the data. The publish/subscribe mechanism is fully implemented, so that when the slave database synchronizes the tree anywhere, a channel can be subscribed to and the complete message publication record of the master server can be received. Synchronization is helpful for scalability and data redundancy of read operations.
A distributed lock. Redis is a Java resident Memory Data Grid (In-Memory Data Grid) that is built on the Redis basis. Redison implements a distributed Lock. Rediston also provides interlocks (MultiLock), read-write locks (ReadWriteLock), fair locks (Fair Lock), red locks (RedLock), semaphores (Semaphore), outdated semaphores (PermitExpirablemap) and lockout (CountDown latch) on a distributed Lock basis.
FIG. 1 is a schematic diagram of a malicious resource access in the prior art. Aiming at the problems that client resources are maliciously accessed and traffic is stolen and brushed, and the like, the invention aims to meet the demand of client timestamp anti-theft chain, and the invention carries out real-time accumulation statistics on some domain names, and limits the current of a general domain name white list reaching a threshold value at any time point so as to provide public service, can carry out delay batch processing under the condition of meeting the reliability and real-time requirements of log acquisition, and can seal and disable client IP in real time, thereby reducing the economic loss of end users in some scenes.
The architecture of the independent user number calculation method under the continuous time constraint is shown in fig. 2, a log collector sends logs needing to be subjected to anti-hotlink enabling to a data processing center queue (namely a message queue in fig. 2) of a content delivery network CDN, a statistical analysis program of the data processing center analyzes IP information of a client Internet protocol, records needing to be reported are sent to a reporting queue, a data reporting program (namely a link statistical reporting program in fig. 2) reports the reporting queue to a control console (namely an anti-hotlink control console in fig. 2), the control console analyzes the reported records, analyzes a service of a white list and seals and bans IP information in a non-white list.
In a first aspect, the present invention provides a method for calculating the number of independent users under continuous time constraint, as shown in fig. 3, specifically, the method includes steps S301 to S304.
S301, acquiring log data, and transmitting the log which needs to be subjected to anti-hotlink enabling to a data processing center queue of a content delivery network CDN;
as an alternative example, collecting log data includes:
the log collector collects the edge log files, packages the log data into a hypertext transfer protocol (HTTP) request according to the sequence of the generation time, and sends the HTTP request to a load balancer of the configured edge log access cluster.
As an alternative example, sending the log that requires hotlink enabling to the data processing center queue of the CDN includes:
and the shunting program of the processing center sends the log which needs to be energized by the anti-theft chain to a CDN data processing center queue, and if the sending fails, retries according to a preset strategy until the sending is successful.
For example, a log collector (such as rsyslog in fig. 2) collects the log files of the edge, encapsulates the log data into an HTTP request according to the priority of the data generation time, and sends the HTTP request to the load balancer of the configured edge log access cluster. And then the shunting program of the data processing center sends the log which needs to be energized by the anti-theft chain to the CDN data processing center queue, and retries the log according to a certain strategy until the sending is successful.
As an alternative example, the retry according to a certain policy may be a preset retry policy, for example, may be: if the transmission is unsuccessful, continuously retrying the transmission for N times, wherein N is a configurable value which is greater than or equal to 1, and if the transmission fails in the continuous N times of retrying, stopping the transmission and sending out an alarm. It is also possible that: if the transmission is unsuccessful, retry M groups continuously, retry N times within each group, pause S seconds between each group, M is an integer greater than or equal to 2, N is an integer greater than or equal to 1, and S is the length of pause time between each group of attempts. M, N, S are all configurable. Other user-set policies are also possible.
S302, the data processing center analyzes the IP information of the client Internet protocol and sends the record to be reported to a reporting queue;
as an alternative example, the data processing center analyzing the client internet protocol IP information includes:
splitting the uniform resource locator URL into a user authentication identifier and a client unique identifier;
counting the number of client IP and service time splices according to the split dimension;
the split latitude is calculated into a character string with 32 bits according to MD5, the character string is stored into a database Redis as a group of key word storage values, locking operation is carried out during storage, and when a lock is obtained, whether the same key word exists is inquired from the database;
if the keyword does not exist, carrying out data analysis and conversion on a value corresponding to the keyword, combining the aggregation latitude into the recorded keyword, splicing and assembling the client IP and the service time into an object set, sequencing and intercepting the set, and when the size of the set is larger than the preset length, intercepting only the preset length and storing the set into a database, otherwise, storing the whole set into the database;
if the keyword exists, a value corresponding to the keyword is taken out, the aggregation latitude is formed into the recorded keyword, the client IP and the service time are spliced and assembled into an object set, the object set and the set in the database are combined, sorting and intercepting are carried out, when the size of the set is larger than the preset length, only the preset length is intercepted and stored in the database, and otherwise, the whole set is stored in the database;
wherein the aggregate dimension comprises one or a combination of: IP address, port, resource address, user authentication information and limit number of times.
As an alternative example, the data processing center analyzing the client internet protocol IP information may further include:
and when the length of the white list set is greater than or equal to the preset length, sending the data to a message queue of a report library, and storing the data in a database, and if the length of the white list set is less than the preset length, storing the data in the database only.
For example, one example of step S302 is as follows:
(1) Splitting url into sign (user authentication identifier), accountId (client unique identifier), host (host), IP (IP address), rlimit (set restriction size, i.e. length size of white list).
(2) Counting the number of client IP and business time splicing according to the dimension of split.
(3) And performing MD5 operation on the split latitude to obtain a 32-bit character string as a set of key storage values Redis, performing locking operation through a redeposition sdk re-entrant lock of the Redis during process storage, simultaneously preventing transaction problems during calculation by other statistical analysis programs, and inquiring whether the same key exists in the Redis when the lock is acquired.
(4) And (3) the key is not existed, and the value corresponding to the key in the batch is subjected to data analysis and conversion. As an alternative example, the transformed data structure is as follows:
(5) And combining the aggregation latitudes into a key of the record, assembling a set of beans (objects) by analyzing the client IP and the business time splicing, sequencing the set, and intercepting. When the size of the set is larger than the preset length rlimit, only the rlimit length is intercepted, otherwise, the whole union set is stored in the Redis. When the size of the aggregate length is larger than or equal to the preset length rlimit, the data is sent to a message queue of a report library, and the message queue is stored to Redis. If only Redis is saved when the aggregate length is less than the predetermined length rlimit.
(6) When the key exists in the Reids, the value corresponding to the key is taken out, and the white list in the batch which is just processed and the white list in the Redis are subjected to data union, sequencing and interception. When the size of the union is larger than the preset length rlimit, only the length of the rlimit is taken out, otherwise, the whole union is stored in the Redis and the record is sent to the report queue, otherwise, only the Redis is stored.
S303, the data reporting program reports the reporting queue to the console;
as an alternative example, the data reporting procedure reporting the reporting queue to the console includes:
subscribing a report queue by a data report program, packaging records reported by a statistical analysis program into HTTP requests, and reporting to a console;
wherein the package send request is triggered when a lot size condition or an interval time condition is satisfied.
For example, the data reporting program subscribes to the reporting queue, packages the records reported by the statistical analysis program into an HTTP request, reports the HTTP request to the console, and the reporting policy of the reporting program is composed of two conditions of a batch size and an interval time, and packages and sends the request when which condition is triggered first, that is, only one or two of the batch size condition or the interval condition are triggered simultaneously, and packages and sends the request.
S304, the control console analyzes the reported records, analyzes the service of the white list, and seals and bans the IP information in the non-white list.
As an optional example, when the console receives the report records packaged by the report program, the console analyzes the report records, performs service analysis on the whitelist list, and performs whitelist registration on the gateway, wherein the blocking duration is determined by an expiration time stamp expireTs in the report records;
request refusals are given to records not in the whitelist.
For a better explanation of the method for calculating the number of independent users under the continuous time constraint of the present invention, a specific example is given below:
(1) The log acquisition service uses rsyslog of an open source, reads the message from the message queue service, assembles the message into an HTTP request body and sends the HTTP request body to the message queue;
(2) The message queue uses Kafka (Kafka is a high throughput distributed publish-subscribe message system that can handle all action flow data of consumers in websites). The Kafka cluster size is 60 instance 180 partitions.
(3) The statistical analysis program and the reporting program are deployed in two data processing centers (e.g., guizhou and inner Mongolia) respectively under continuous time constraint, the Kafka consumption rate is 10K, the number of 60 executors (executives), each executure is 3 cores (3 core), the processing batch size is 1 minute, and the processing rate per minute is 100000K++ TPS.
(4) The Redis cluster adopts a cluster deployment mode of 10 (masset) +10 (salve), namely 10 main clusters and 10 auxiliary clusters, the reading and writing speed per second reaches 60 ten thousand qps/S, and the re-disssion can reenter the lock configuration time is 30S.
(5) In the implementation flow, configuration is only carried out by realizing rlimit (limit white list number), expaties (expiration time), channels (domain names), batch size, duration (reporting interval time) and the like of different domain names in a page.
By using the method, the stability and performance of data acquisition are ensured by using the high-availability big data analysis component, the consumption capacity of the data is improved, and backlog of the data is prevented.
The statistical analysis stage is deployed in a cluster mode, real-time transverse expansion is realized, and when the consumption performance is insufficient, the consumption capacity of data is improved by adding the consumption program of statistical analysis or improving the execution number core number of a single statistical analysis consumption program, so that the backlog of the data is prevented, and the consumption capacity reaches tens of millions of TPS/S.
The intermediate buffer can use Redis to prevent the loss of intermediate data calculated in real time and improve the reading and writing capability. The distribution type calculation carries out the re-locking of the white list statistics by using the Redis, so that the atomicity, the uniqueness and the safety of the data are ensured.
In a second aspect, another embodiment of the present invention discloses an independent user number calculating apparatus under continuous time constraint, as shown in fig. 4, including:
the log module 401 is configured to collect log data, and send the log to be subjected to anti-hotlink enabling to a data processing center queue of the content delivery network CDN;
the data processing center 402 is configured to analyze the client internet protocol IP information and send the record to be reported to the reporting queue;
a reporting module 403 configured to report the reporting queue to a console;
the console 404 is configured to parse the reported record, parse the whitelist for service, and register the whitelist for the gateway.
As an alternative example, the log module 401 is further configured to encapsulate the log data into a hypertext transfer protocol HTTP request according to the sequence of the generation time, and send the HTTP request to the load balancer of the configured edge log access cluster.
As an alternative example, the splitting program of the data processing center 402 sends the log that needs to be hotlink enabled to the CDN data processing center queue, and if the sending fails, retries according to a predetermined policy until the sending is successful.
As an alternative example, the data processing center 402 is further configured to:
splitting the uniform resource locator URL into a user authentication identifier and a client unique identifier; counting the number of client IP and service time splices according to the split dimension;
the split latitude is calculated into a character string with 32 bits according to MD5, the character string is stored into a database Redis as a group of key word storage values, locking operation is carried out during storage, and when a lock is obtained, whether the same key word exists is inquired from the database;
if the keyword does not exist, carrying out data analysis and conversion on a value corresponding to the keyword, combining the aggregation latitude into the recorded keyword, splicing and assembling the client IP and the service time into an object set, sequencing and intercepting the set, and when the size of the set is larger than the preset length, intercepting only the preset length and storing the set into a database, otherwise, storing the whole set into the database;
if the keyword exists, a value corresponding to the keyword is taken out, the aggregation latitude is formed into the recorded keyword, the client IP and the service time are spliced and assembled into an object set, the object set and the set in the database are combined, sorting and intercepting are carried out, when the size of the set is larger than the preset length, only the preset length is intercepted and stored in the database, and otherwise, the whole set is stored in the database;
wherein the aggregate dimension comprises one or a combination of: IP address, port, resource address, user authentication information and limit times;
and when the length of the white list set is greater than or equal to the preset length, sending the data to a message queue of a report library, and storing the data in a database, and if the length of the white list set is less than the preset length, storing the data in the database only.
As an optional example, the reporting module 403 is further configured to subscribe to a reporting queue, package the record reported by the statistical analysis program into an HTTP request, and report the HTTP request to the console;
wherein the package send request is triggered when a lot size condition or an interval time condition is satisfied.
As an alternative example, console 404 is further configured to:
when receiving the report records packed by the report program, analyzing the report records, analyzing the service of the white list, and registering the white list of the gateway, wherein the blocking duration is determined by an expiration time stamp expireTs in the report records;
request refusals are given to records not in the whitelist.
In a third aspect, the present invention proposes an electronic device, as shown in fig. 5, including: a memory 502 and one or more processors 501 and a user interface 503.
The memory 501 is used for storing a computer program;
the user interface 503 is configured to interact with a user;
the memory 502 has stored therein one or more applications adapted to be executed by the one or more processors to implement the method of calculating the number of individual users under the continuous time constraint of the first aspect.
As shown in FIG. 5, the processor 502 is coupled to the memory 502, such as via a bus interface.
The structure of the electronic device is not limited to the embodiment of the present invention.
The processor 501 may be a CPU, general purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules and circuits described in connection with this disclosure. The processor 501 may also be a combination that implements computing functionality, such as a combination comprising one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
A bus may include a path that communicates information between the components. The bus may be a PCI bus or an EISA bus, etc. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one bus or one type of bus is shown in FIG. 5.
Memory 502 may be, but is not limited to, a ROM or other type of static storage device, a RAM or other type of dynamic storage device, which may store static information and instructions, an EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compact, laser, optical, digital versatile, blu-ray, etc.), magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
In a fourth aspect, the present invention proposes a computer readable storage medium having stored thereon a computer program capable of being loaded and executed by a processor with the method of calculating the number of independent users under continuous time constraints of the first aspect.
While the applicant has described and illustrated the embodiments of the present invention in detail with reference to the drawings, it should be understood by those skilled in the art that the above embodiments are only preferred embodiments of the present invention, and the detailed description is only for the purpose of helping the reader to better understand the spirit of the present invention, and not to limit the scope of the present invention, but any improvements or modifications based on the spirit of the present invention should fall within the scope of the present invention.
Finally, it should be noted that the above embodiments are merely for illustrating the technical solution of the embodiments of the present invention, and are not limiting. Although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the embodiments of the invention, and any changes and substitutions that would be apparent to one skilled in the art are intended to be included within the scope of the present invention.
Claims (10)
1. A method for calculating the number of independent users under continuous time constraint, comprising:
collecting log data, and sending the log which needs to be subjected to anti-hotlink enabling to a data processing center queue of a content delivery network CDN;
the data processing center analyzes the IP information of the client Internet protocol and sends the record to be reported to a reporting queue;
the data reporting program reports the reporting queue to a control console;
and the control console analyzes the reported records, analyzes the service of the white list and seals and bans the IP information in the non-white list.
2. The method of claim 1, wherein collecting log data comprises:
the log collector collects the edge log files, packages the log data into a hypertext transfer protocol (HTTP) request according to the sequence of the generation time, and sends the HTTP request to a load balancer of the configured edge log access cluster.
3. The method for calculating the number of independent users under continuous time constraint according to claim 1, wherein the step of sending the log to be given the burglar-link enable to the data processing center queue of the CDN comprises:
and the shunting program of the processing center sends the log which needs to be energized by the anti-theft chain to a CDN data processing center queue, and if the sending fails, retries according to a preset strategy until the sending is successful.
4. The method of claim 1, wherein the data processing center analyzing the client internet protocol IP information comprises:
splitting the uniform resource locator URL into a user authentication identifier and a client unique identifier;
counting the number of client IP and service time splices according to the split dimension;
the split latitude is calculated into a character string with 32 bits according to MD5, the character string is stored into a database Redis as a group of key word storage values, locking operation is carried out during storage, and when a lock is obtained, whether the same key word exists is inquired from the database;
if the keyword does not exist, carrying out data analysis and conversion on a value corresponding to the keyword, combining the aggregation latitude into the recorded keyword, splicing and assembling the client IP and the service time into an object set, sequencing and intercepting the set, and when the size of the set is larger than the preset length, intercepting only the preset length and storing the set into a database, otherwise, storing the whole set into the database;
if the keyword exists, a value corresponding to the keyword is taken out, the aggregation latitude is formed into the recorded keyword, the client IP and the service time are spliced and assembled into an object set, the object set and the set in the database are combined, sorting and intercepting are carried out, when the size of the set is larger than the preset length, only the preset length is intercepted and stored in the database, and otherwise, the whole set is stored in the database;
wherein the aggregate dimension comprises one or a combination of: IP address, port, resource address, user authentication information and limit number of times.
5. The method for calculating the number of independent users under continuous time constraint according to claim 4, further comprising:
and when the length of the white list set is greater than or equal to the preset length, sending the data to a message queue of a report library, and storing the data in a database, and if the length of the white list set is less than the preset length, storing the data in the database only.
6. The method of claim 1, wherein the data reporting program reporting the report queue to a console comprises:
subscribing a report queue by a data report program, packaging records reported by a statistical analysis program into HTTP requests, and reporting to a console;
wherein the package send request is triggered when a lot size condition or an interval time condition is satisfied.
7. The method for calculating the number of independent users under continuous time constraint according to claim 6, wherein the control console analyzes the reported record, performs service analysis on the whitelist, and performs blocking on the IP information in the non-whitelist, comprising:
when the control console receives the report records packed by the report program, analyzing the report records, carrying out service analysis on a white list, and registering the white list for the gateway, wherein the blocking duration is determined by an expiration time stamp expireTs in the report records;
request refusals are given to records not in the whitelist.
8. An independent user number calculation apparatus under continuous time constraint, the apparatus comprising:
the log module is configured to collect log data and send the log which needs to be subjected to anti-hotlink enabling to a data processing center queue of the content delivery network CDN;
the data processing center is configured to analyze the IP information of the client Internet protocol and send the record to be reported to the reporting queue;
the reporting module is configured to report the reporting queue to a control console;
and the control console is configured to analyze the reported records, analyze the service of the white list and register the white list of the gateway.
9. An independent user number computing device under continuous time constraints, comprising a memory, a processor, and a user interface;
the memory is used for storing a computer program;
the user interface is used for realizing interaction with a user;
the processor being configured to read a computer program in the memory, the processor implementing the method for calculating the number of independent users under continuous time constraint according to one of claims 1 to 7 when the computer program is executed.
10. A processor-readable storage medium, characterized in that the processor-readable storage medium stores a computer program, which when executed by the processor implements the method of calculating the number of independent users under continuous time constraint according to one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311713501.XA CN117834204A (en) | 2023-12-13 | 2023-12-13 | Independent user number calculation method and device under continuous time constraint |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311713501.XA CN117834204A (en) | 2023-12-13 | 2023-12-13 | Independent user number calculation method and device under continuous time constraint |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117834204A true CN117834204A (en) | 2024-04-05 |
Family
ID=90510628
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311713501.XA Pending CN117834204A (en) | 2023-12-13 | 2023-12-13 | Independent user number calculation method and device under continuous time constraint |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117834204A (en) |
-
2023
- 2023-12-13 CN CN202311713501.XA patent/CN117834204A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11640474B2 (en) | Method and apparatus for operating database | |
| JP6716727B2 (en) | Streaming data distributed processing method and apparatus | |
| CN111831748B (en) | Data synchronization method, device and storage medium | |
| WO2017048924A1 (en) | Distributed data processing method and system | |
| US20020120697A1 (en) | Multi-channel messaging system and method | |
| WO2018044828A1 (en) | System and method for creating time-accurate events streams | |
| CN111831458B (en) | High-concurrency high-decoupling data processing method and data center system | |
| CN112632129A (en) | Code stream data management method, device and storage medium | |
| US20230370285A1 (en) | Block-chain-based data processing method, computer device, computer-readable storage medium | |
| CN101442558A (en) | Method and system for providing index service for P2SP network | |
| EP3803616A1 (en) | Change notifications for object storage | |
| CN111698126B (en) | Information monitoring method, system and computer readable storage medium | |
| CN114090555A (en) | AIS data processing method and system | |
| CN117834204A (en) | Independent user number calculation method and device under continuous time constraint | |
| CN112667586B (en) | Method, system, equipment and medium for synchronizing data based on stream processing | |
| CN113051323B (en) | Water environment big data exchange method | |
| CN103686224A (en) | Method and system for transcoding task obtaining on basis of distributed locks | |
| CN114547214A (en) | Long-link data processing method, device and equipment | |
| US10819622B2 (en) | Batch checkpointing for inter-stream messaging system | |
| WO2022252357A1 (en) | Consensus processing method and apparatus for blockchain network, device, system, and medium | |
| CN112650915B (en) | Data interaction method and device based on real-time query | |
| CN113179333B (en) | File transmission and real-time computing system and method for block chain | |
| CN112256446B (en) | Kafka message bus control method and system | |
| CN113961149B (en) | Polymorphic data storage system and method for electric power information system | |
| CN101883118A (en) | Digital signature method for mass data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |