CN112764937A - Access control method and device, computer equipment and storage medium - Google Patents
Access control method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112764937A CN112764937A CN202110124822.0A CN202110124822A CN112764937A CN 112764937 A CN112764937 A CN 112764937A CN 202110124822 A CN202110124822 A CN 202110124822A CN 112764937 A CN112764937 A CN 112764937A
- Authority
- CN
- China
- Prior art keywords
- access request
- request
- time
- access
- reply
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an access amount control method, an access amount control device, computer equipment and a storage medium, which relate to the big data technology and comprise the steps of obtaining a request type and request time corresponding to a current access request; if the request time corresponding to the current access request is in the access active time interval and the current access request is a real-time reply request, generating reply data corresponding to the current access request, and sending the reply data to the user side; if the current access request is a non-real-time reply request, sending the current access request to a distributed log system for storage; and consuming the access request from the distributed log system by adopting a single thread mode, and correspondingly generating delayed reply data according to the access request obtained by consumption. The peak clipping and valley filling of the access requests are realized, the processing of the non-real-time requests is effectively delayed in the access active time interval, most of resources of the server can be used for processing the real-time requests in a centralized mode, the processing pressure of the server is reduced, and the server cannot be easily crashed.
Description
Technical Field
The invention relates to the technical field of data processing of big data, in particular to an access amount control method and device, computer equipment and a storage medium.
Background
Currently, system integration of data access and interaction within a server is essential and is a necessary capability of a distributed system. However, as the number of users of the system increases, the data of the system also grows exponentially, especially in hundreds of millions during the time period when the users are active during one day. Under these access scenarios, the original https request type data transmission interaction exerts a great pressure on load balancing and servers, which causes frequent resource scheduling for the servers and reduced utilization efficiency, and even causes the servers to run out of resources.
Disclosure of Invention
The embodiment of the invention provides an access amount control method and device, computer equipment and a storage medium, and aims to solve the problem that a server is easy to exhaust and crash resources of the server due to the fact that a server replies massive access requests in a short time in the prior art.
In a first aspect, an embodiment of the present invention provides an access amount control method, including:
acquiring a historical access request statistical data set stored locally, and acquiring an access active time interval according to the historical access request statistical data set;
if a current access request sent by a user side is received, acquiring a request type and request time corresponding to the current access request;
if the request time corresponding to the current access request is within the access active time interval, judging whether the current access request is a real-time reply request or a non-real-time reply request;
if the current access request is a real-time reply request, generating reply data corresponding to the current access request, and sending the reply data to a user side;
if the current access request is a non-real-time reply request, sending the current access request to a distributed log system for storage; and
and consuming the access request from the distributed log system by adopting a single thread mode, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
In a second aspect, an embodiment of the present invention provides an access amount control apparatus, including:
the access active time interval acquisition unit is used for acquiring a locally stored historical access request statistical data set and acquiring an access active time interval according to the historical access request statistical data set;
the current access request analyzing unit is used for acquiring a request type and request time corresponding to a current access request if the current access request sent by a user side is received;
a current access request type determining unit, configured to determine whether the current access request is a real-time reply request or a non-real-time reply request if a request time corresponding to the current access request is within the access active time interval;
the first reply unit is used for generating reply data corresponding to the current access request and sending the reply data to the user side if the current access request is a real-time reply request;
the request forwarding unit is used for sending the current access request to a distributed log system for storage if the current access request is a non-real-time reply request; and
and the second reply unit is used for consuming the access request from the distributed log system in a single thread mode, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the access amount control method according to the first aspect when executing the computer program.
In a fourth aspect, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the access amount control method according to the first aspect.
The embodiment of the invention provides an access amount control method, an access amount control device, computer equipment and a storage medium, wherein if a current access request sent by a user side is received, a request type and request time corresponding to the current access request are obtained; if the request time corresponding to the current access request is within the access active time interval, judging whether the current access request is a real-time reply request or a non-real-time reply request; if the current access request is a real-time reply request, generating reply data of the current access request, and sending the reply data to the user side; if the current access request is a non-real-time reply request, sending the current access request to a distributed log system for storage; and consuming the access request from the distributed log system by adopting a single thread mode, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side. The peak clipping and valley filling of the access requests are realized, the processing of the non-real-time requests is effectively delayed in the access active time interval, most of resources of the server can be used for processing the real-time requests in a centralized mode, the processing pressure of the server is reduced, and the server cannot be easily crashed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of an access point control method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of an access control method according to an embodiment of the present invention;
FIG. 3 is a schematic block diagram of an access control apparatus provided by an embodiment of the present invention;
FIG. 4 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of an access control method according to an embodiment of the present invention; fig. 2 is a schematic flowchart of an access control method according to an embodiment of the present invention, where the access control method is applied to a server and is executed by application software installed in the server.
As shown in fig. 2, the method includes steps S110 to S160.
S110, obtaining a locally stored historical access request statistical data set, and obtaining an access active time interval according to the historical access request statistical data set.
In this embodiment, the user side may initiate a https or other type request to the server, so as to obtain corresponding return data. For example, a user accesses a portal to read information or accesses various forums, and the like, and during the interaction between the user and the server, the process of continuously sending access requests and receiving corresponding return data is performed. Because the number of the user terminals connected with the server is more, the access requests in the server have a plurality of hours with the access volume in a peak state in a statistical interval of one day, and the access volumes in other most time are less or more gentle.
In order to effectively relieve the service pressure of the server in the peak access time period, the access active time interval can be obtained by analysis. Since the historical access request statistical data set is locally stored in the server, the historical access request statistical data set may be a statistical data set of the last month, a statistical data set of the last quarter, or a statistical data set of the last year, and the access active time interval obtained through analysis is more accurate if the data amount in the statistical data set is larger.
In one embodiment, step S110 includes:
acquiring historical access request statistical data subsets respectively corresponding to the historical access request statistical data sets every day;
acquiring historical access request distribution graphs respectively corresponding to the historical access request statistical data subsets according to access time points corresponding to the historical access requests in the historical access request statistical data subsets; the historical access request distribution curve takes a time axis as a horizontal axis and takes the number of the access requests as a vertical axis;
acquiring a part of each historical access request distribution curve graph, which is positioned above a preset access amount threshold curve, to obtain target distribution curve graphs corresponding to the historical access request distribution curve graphs respectively;
and acquiring time intervals corresponding to the target distribution curves, respectively averaging the starting time and the ending time of each time interval to obtain corresponding starting time average values and ending time average values, and taking the starting time average values as starting time points and the ending time average values as ending time points as access active time intervals.
In this embodiment, in order to more accurately obtain the access active time interval, a historical access request statistical data set locally stored by the server may be obtained first, and then a historical access request statistical data subset corresponding to each day is obtained with 24 hours corresponding to a natural day as a statistical period.
After the statistical data subset of the historical access requests corresponding to each day is obtained, each historical access request corresponds to a specific access time, so that the distribution curve graph of the historical access requests corresponding to each day can be obtained by taking a time axis as a horizontal axis and taking the number of the access requests as a vertical axis. For example, in 2018, 11, 16: the number of historical access requests corresponding to the time point 00 is 10 ten thousand, the set access amount threshold curve is that y is 30000, and obviously 16 days 11 and 11 months 11 and 11 in 2018: the number of the historical access requests corresponding to the time point 00 is above the value of y 30000 in the historical access request distribution graph of the current day. In this way, all data points above the preset threshold curve of the access amount are found, so that the target distribution curves respectively corresponding to the historical access request distribution curves can be obtained.
Because each target distribution curve graph can know the corresponding time interval from the horizontal axis, each time interval corresponds to a starting time and a terminating time, the average value of the starting time and the average value of the terminating time of a plurality of time intervals are counted, and the access active time interval can be obtained based on historical data. By the analysis processing mode based on the historical access request distribution curve graph, the access active time interval can be obtained more quickly and accurately.
S120, if a current access request sent by a user side is received, obtaining a request type and request time corresponding to the current access request.
In this embodiment, if a server receives a current access request sent by a certain client, in order to effectively reduce the load of the server, a request time and a request type corresponding to the current access request may be obtained first. When the request type and the request time corresponding to the current access request are obtained, the request content in the current access request is analyzed. For example, if the current access request is an https request, the specific content of the https request may be parsed to obtain the corresponding request type and request time.
The request type corresponding to the current access request comprises a real-time reply request and a non-real-time reply request; the current access request with the type of the real-time reply request refers to an access request that the server needs to feed back a reply in time, and the current access request with the type of the non-real-time reply request refers to an access request that the server does not need to feed back a reply in time (the non-real-time reply request may be a request that the server returns after a delay for a certain period of time, or has no effect, for example, the server returns after delaying for 8 hours, or returns after delaying for 0.5 hours).
S130, if the request time corresponding to the current access request is within the access active time interval, judging whether the current access request is a real-time reply request or a non-real-time reply request.
In this embodiment, if the request time corresponding to the current access request is within the access active time interval, the server may not send the feedback data corresponding to all access requests in time because the server receives more access requests in unit time. At this time, in order to preferentially feed back an access request with a high real-time requirement, it is necessary to determine whether the current access request is a real-time reply request or a non-real-time reply request.
In an embodiment, step S130 is followed by:
and if the request time corresponding to the current access request is not within the access active time interval, generating reply data corresponding to the current access request, and sending the reply data to the user side.
In this embodiment, if the request time corresponding to the current access request is not within the access active time interval, the server may send feedback data corresponding to all access requests in time if the access requests received by the server within a unit time are not many. At this time, only the reply data corresponding to the current access request needs to be generated, and the reply data is sent to the user side.
And S140, if the current access request is a real-time reply request, generating reply data corresponding to the current access request, and sending the reply data to the user side.
In this embodiment, if the request time corresponding to the current access request is within the access active time interval and the current access request is a real-time reply request, it indicates that the current access request needs the timely feedback data of the server, and at this time, the reply data corresponding to the current access request is still generated, and the reply data is sent to the user side.
Here, an example is described in which a request type sent by a client is a real-time reply request, and actually, a server receives a plurality of current access requests with request types being real-time reply requests, and at this time, the server may concurrently process the request types being the current access requests of the real-time reply requests according to its processing capability, and may also reserve a small portion of system resources for processing the current access requests with request types being non-real-time reply requests in a single thread mode.
S150, if the current access request is a non-real-time reply request, sending the current access request to a distributed log system for storage.
In this embodiment, if the request time corresponding to the current access request is within the access active time interval and the current access request is a non-real-time reply request, at this time, in order to effectively reduce the data processing pressure of the server, the current access request may be first sent to the distributed log system for storage. The distributed log system, namely the Kafka system, provides distributed extensible message queue service.
Because the non-real-time reply requests forwarded to the distributed log system by the server have corresponding forwarding time, after the current access requests of a plurality of non-real-time reply request types are forwarded to the distributed log system, the current access requests are arranged in the distributed log system according to the sequence of the forwarding time (namely, the principle of first-in first-out is satisfied). Because the server takes the distributed log system as a cache container for non-real-time reply requests, the data processing pressure of the server is effectively reduced.
S160, consuming the access request from the distributed log system by adopting a single thread mode, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to a user side.
In this embodiment, since a small portion of system resources are reserved in the server for processing the current access request with the request type of non-real-time reply request in the single thread mode, there are several ways to delay the reply of the current access request.
As a first specific embodiment of step S160, it includes:
and if the time interval between the current single-thread processing time and the previous single-thread processing time is equal to a preset single-thread processing time threshold value, consuming the access request from the distributed log system, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
In this embodiment, as a first embodiment in which the server consumes the access requests from the distributed log system, in order to effectively control the frequency of the current access requests, of which the request types processed by the server are non-real-time reply requests, in the access active time interval, a first way is to limit the time interval in which the server consumes the access requests from the distributed log system. For example, if the single thread processing time threshold is set to 1s, the server will consume 1 access request from the distributed log system every 1s and correspondingly feed back the access request. If the time length corresponding to the access active time interval is 4h, namely within the 4h, the server consumes 14400 access requests at most and correspondingly feeds back the access requests, and the other rest of system resources of the server process the current access requests of the request in parallel, wherein the type of the current access requests is the type of the real-time reply request.
As a second specific embodiment of step S160, it includes:
and if the current system time is not in the access active time interval, sequentially consuming the access requests from the distributed log system, correspondingly generating delayed reply data according to the access requests obtained by consumption, and sending the delayed reply data to the user side.
In this embodiment, as a second embodiment in which the server consumes the access requests from the distributed log system, all the non-real-time reply requests whose request time is within the access active time interval may be temporarily stored in the distributed log system in the order of forwarding time, and the non-real-time reply requests may be temporarily processed until the current system time exceeds the termination time point of the access active time interval, and at this time, after the non-access active time interval, the resources of the server are sufficient, the access requests may be sequentially consumed from the distributed log system to correspond to the feedback.
However, no matter the first embodiment or the second embodiment of step S160 is adopted, a situation may be encountered where the server and the distributed log system are not in the same local area network, which may result in a situation where the server and the distributed log system may be in network connection, but the server may not normally consume the access request from the distributed log system. In order to avoid this problem, step S160 is preceded by:
judging whether the distributed log system and the distributed log system are in the same internal local area network or not;
if the distributed log system is not in the same internal local area network as the distributed log system, acquiring an external network IP port of the distributed log system, and establishing connection with the distributed log system through the external network IP port;
and if the internal network IP port of the distributed log system is in the same internal local area network as the distributed log system, acquiring the internal network IP port of the distributed log system, and establishing connection with the distributed log system through the internal network IP port.
In this embodiment, since the technical solution is described in terms of a server, it is determined whether the server and the distributed log system are in the same internal local area network. For example, the distributed logging system is in a lan and the server is in B lan, both of which can ping (ping is an instruction to test a network connection), but the server cannot consume an access request from the distributed logging system.
At this time, the above problem can be solved by using the Kafka internal and external network separation technology, that is, two externally output IP ports are configured for the distributed log system, one is an internal network IP port, and the other is an external network IP port. If the server and the distributed log system are not in the same internal local area network, connection is established with the distributed log system through the external network IP port; and if the server and the distributed log system are in the same internal local area network, establishing connection with the distributed log system through the intranet IP port. By the processing mode, the server can normally consume the access request from the distributed log system.
The method realizes peak clipping and valley filling of the access request, effectively delays processing of the non-real-time request in the access active time interval, can enable most resources of the server to be used for centralized processing of the real-time request, reduces the processing pressure of the server, and cannot cause the server to be easily down.
An embodiment of the present invention further provides an access amount control device, where the access amount control device is configured to execute any embodiment of the foregoing access amount control method. Specifically, referring to fig. 3, fig. 3 is a schematic block diagram of an access amount control apparatus according to an embodiment of the present invention. The access amount control device 100 may be disposed in a server.
As shown in fig. 3, the access amount control apparatus 100 includes: an access active time interval obtaining unit 110, a current access request analyzing unit 120, a current access request type determining unit 130, a first replying unit 140, a request forwarding unit 150, and a second replying unit 160.
An access active time interval obtaining unit 110, configured to obtain a locally stored historical access request statistical data set, and obtain an access active time interval according to the historical access request statistical data set.
In this embodiment, the user side may initiate a https or other type request to the server, so as to obtain corresponding return data. For example, a user accesses a portal to read information or accesses various forums, and the like, and during the interaction between the user and the server, the process of continuously sending access requests and receiving corresponding return data is performed. Because the number of the user terminals connected with the server is more, the access requests in the server have a plurality of hours with the access volume in a peak state in a statistical interval of one day, and the access volumes in other most time are less or more gentle.
In order to effectively relieve the service pressure of the server in the peak access time period, the access active time interval can be obtained by analysis. Since the historical access request statistical data set is locally stored in the server, the historical access request statistical data set may be a statistical data set of the last month, a statistical data set of the last quarter, or a statistical data set of the last year, and the access active time interval obtained through analysis is more accurate if the data amount in the statistical data set is larger.
In an embodiment, the access active time interval acquisition unit 110 includes:
a history subset obtaining unit, configured to obtain history access request statistical data subsets corresponding to the history access request statistical data sets respectively every day;
a historical access request statistical data subset acquiring unit, configured to acquire historical access request distribution graphs corresponding to the historical access request statistical data subsets according to access time points corresponding to each historical access request in the historical access request statistical data subsets; the historical access request distribution curve takes a time axis as a horizontal axis and takes the number of the access requests as a vertical axis;
a target distribution curve graph obtaining unit, configured to obtain a portion, located above a preset access amount threshold curve, of each historical access request distribution curve graph, to obtain target distribution curve graphs corresponding to the historical access request distribution curve graphs, respectively;
and the time interval averaging unit is used for acquiring the time intervals corresponding to the target distribution graphs, averaging the starting time and the ending time of each time interval respectively to obtain corresponding starting time average values and ending time average values, and taking the time intervals with the starting time average values as the starting time points and the ending time average values as the ending time points as the access active time intervals.
In this embodiment, in order to more accurately obtain the access active time interval, a historical access request statistical data set locally stored by the server may be obtained first, and then a historical access request statistical data subset corresponding to each day is obtained with 24 hours corresponding to a natural day as a statistical period.
After the statistical data subset of the historical access requests corresponding to each day is obtained, each historical access request corresponds to a specific access time, so that the distribution curve graph of the historical access requests corresponding to each day can be obtained by taking a time axis as a horizontal axis and taking the number of the access requests as a vertical axis. For example, in 2018, 11, 16: the number of historical access requests corresponding to the time point 00 is 10 ten thousand, the set access amount threshold curve is that y is 30000, and obviously 16 days 11 and 11 months 11 and 11 in 2018: the number of the historical access requests corresponding to the time point 00 is above the value of y 30000 in the historical access request distribution graph of the current day. In this way, all data points above the preset threshold curve of the access amount are found, so that the target distribution curves respectively corresponding to the historical access request distribution curves can be obtained.
Because each target distribution curve graph can know the corresponding time interval from the horizontal axis, each time interval corresponds to a starting time and a terminating time, the average value of the starting time and the average value of the terminating time of a plurality of time intervals are counted, and the access active time interval can be obtained based on historical data. By the analysis processing mode based on the historical access request distribution curve graph, the access active time interval can be obtained more quickly and accurately.
The current access request parsing unit 120 is configured to, if a current access request sent by a user is received, obtain a request type and a request time corresponding to the current access request.
In this embodiment, if a server receives a current access request sent by a certain client, in order to effectively reduce the load of the server, a request time and a request type corresponding to the current access request may be obtained first. When the request type and the request time corresponding to the current access request are obtained, the request content in the current access request is analyzed. For example, if the current access request is an https request, the specific content of the https request may be parsed to obtain the corresponding request type and request time.
The request type corresponding to the current access request comprises a real-time reply request and a non-real-time reply request; the current access request with the type of the real-time reply request refers to an access request that the server needs to feed back a reply in time, and the current access request with the type of the non-real-time reply request refers to an access request that the server does not need to feed back a reply in time (the non-real-time reply request may be a request that the server returns after a delay for a certain period of time, or has no effect, for example, the server returns after delaying for 8 hours, or returns after delaying for 0.5 hours).
A current access request type determining unit 130, configured to determine that the current access request is a real-time reply request or a non-real-time reply request if the request time corresponding to the current access request is within the access active time interval.
In this embodiment, if the request time corresponding to the current access request is within the access active time interval, the server may not send the feedback data corresponding to all access requests in time because the server receives more access requests in unit time. At this time, in order to preferentially feed back an access request with a high real-time requirement, it is necessary to determine whether the current access request is a real-time reply request or a non-real-time reply request.
In one embodiment, the access amount control apparatus 100 further includes:
and the third reply unit is used for generating reply data corresponding to the current access request and sending the reply data to the user side if the request time corresponding to the current access request is not within the access active time interval.
In this embodiment, if the request time corresponding to the current access request is not within the access active time interval, the server may send feedback data corresponding to all access requests in time if the access requests received by the server within a unit time are not many. At this time, only the reply data corresponding to the current access request needs to be generated, and the reply data is sent to the user side.
The first reply unit 140 is configured to generate reply data corresponding to the current access request if the current access request is a real-time reply request, and send the reply data to the user side.
In this embodiment, if the request time corresponding to the current access request is within the access active time interval and the current access request is a real-time reply request, it indicates that the current access request needs the timely feedback data of the server, and at this time, the reply data corresponding to the current access request is still generated, and the reply data is sent to the user side.
Here, an example is described in which a request type sent by a client is a real-time reply request, and actually, a server receives a plurality of current access requests with request types being real-time reply requests, and at this time, the server may concurrently process the request types being the current access requests of the real-time reply requests according to its processing capability, and may also reserve a small portion of system resources for processing the current access requests with request types being non-real-time reply requests in a single thread mode.
And a request forwarding unit 150, configured to send the current access request to a distributed log system for storage if the current access request is a non-real-time reply request.
In this embodiment, if the request time corresponding to the current access request is within the access active time interval and the current access request is a non-real-time reply request, at this time, in order to effectively reduce the data processing pressure of the server, the current access request may be first sent to the distributed log system for storage. The distributed log system, namely the Kafka system, provides distributed extensible message queue service.
Because the non-real-time reply requests forwarded to the distributed log system by the server have corresponding forwarding time, after the current access requests of a plurality of non-real-time reply request types are forwarded to the distributed log system, the current access requests are arranged in the distributed log system according to the sequence of the forwarding time (namely, the principle of first-in first-out is satisfied).
The second replying unit 160 is configured to consume the access request from the distributed log system in a single-thread mode, generate deferred reply data according to the access request obtained by consumption, and send the deferred reply data to the user side.
In this embodiment, since a small portion of system resources are reserved in the server for processing the current access request with the request type of non-real-time reply request in the single thread mode, there are several ways to delay the reply of the current access request.
As a first specific embodiment of the second replying unit 160, it is further configured to:
and if the time interval between the current single-thread processing time and the previous single-thread processing time is equal to a preset single-thread processing time threshold value, consuming the access request from the distributed log system, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
In this embodiment, as a first embodiment in which the server consumes the access requests from the distributed log system, in order to effectively control the frequency of the current access requests, of which the request types processed by the server are non-real-time reply requests, in the access active time interval, a first way is to limit the time interval in which the server consumes the access requests from the distributed log system. For example, if the single thread processing time threshold is set to 1s, the server will consume 1 access request from the distributed log system every 1s and correspondingly feed back the access request. If the time length corresponding to the access active time interval is 4h, namely within the 4h, the server consumes 14400 access requests at most and correspondingly feeds back the access requests, and the other rest of system resources of the server process the current access requests of the request in parallel, wherein the type of the current access requests is the type of the real-time reply request.
As a second specific embodiment of the second replying unit 160, it is further configured to:
and if the current system time is not in the access active time interval, sequentially consuming the access requests from the distributed log system, correspondingly generating delayed reply data according to the access requests obtained by consumption, and sending the delayed reply data to the user side.
In this embodiment, as a second embodiment in which the server consumes the access requests from the distributed log system, all the non-real-time reply requests whose request time is within the access active time interval may be temporarily stored in the distributed log system in the order of forwarding time, and the non-real-time reply requests may be temporarily processed until the current system time exceeds the termination time point of the access active time interval, and at this time, after the non-access active time interval, the resources of the server are sufficient, the access requests may be sequentially consumed from the distributed log system to correspond to the feedback.
However, in the first embodiment or the second embodiment of the second replying unit 160, a situation may be encountered where the server and the distributed log system are not in the same intranet, which may result in a situation where the server and the distributed log system may be in network connection, but the server may not normally consume the access request from the distributed log system. In order to avoid this problem, the access amount control apparatus 100 further includes:
the same local area network judging unit is used for judging whether the distributed log system and the distributed log system are in the same internal local area network or not;
the first connection unit is used for acquiring an external network IP port of the distributed log system if the distributed log system and the distributed log system are not in the same internal local area network, and establishing connection with the distributed log system through the external network IP port;
and the second connection unit is used for acquiring an intranet IP port of the distributed log system if the second connection unit is in the same internal local area network with the distributed log system, and establishing connection with the distributed log system through the intranet IP port.
In this embodiment, since the technical solution is described in terms of a server, it is determined whether the server and the distributed log system are in the same internal local area network. For example, the distributed logging system is in a lan and the server is in B lan, both of which can ping (ping is an instruction to test a network connection), but the server cannot consume an access request from the distributed logging system.
At this time, the above problem can be solved by using the Kafka internal and external network separation technology, that is, two externally output IP ports are configured for the distributed log system, one is an internal network IP port, and the other is an external network IP port. If the server and the distributed log system are not in the same internal local area network, connection is established with the distributed log system through the external network IP port; and if the server and the distributed log system are in the same internal local area network, establishing connection with the distributed log system through the intranet IP port. By the processing mode, the server can normally consume the access request from the distributed log system.
The device realizes peak clipping and valley filling of the access request, effectively delays processing of the non-real-time request in the access active time interval, can enable most resources of the server to be used for centralized processing of the real-time request, reduces the processing pressure of the server, and cannot cause the server to be shut down easily.
The above-described access control means may be implemented in the form of a computer program which is executable on a computer device as shown in fig. 4.
Referring to fig. 4, fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device 500 is a server, and the server may be an independent server or a server cluster composed of a plurality of servers.
Referring to fig. 4, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, may cause the processor 502 to perform an access control method.
The processor 502 is used to provide computing and control capabilities that support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 may be caused to execute the access control method.
The network interface 505 is used for network communication, such as providing transmission of data information. Those skilled in the art will appreciate that the configuration shown in fig. 4 is a block diagram of only a portion of the configuration associated with aspects of the present invention and is not intended to limit the computing device 500 to which aspects of the present invention may be applied, and that a particular computing device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
The processor 502 is configured to run the computer program 5032 stored in the memory to implement the access control method disclosed in the embodiment of the present invention.
Those skilled in the art will appreciate that the embodiment of a computer device illustrated in fig. 4 does not constitute a limitation on the specific construction of the computer device, and that in other embodiments a computer device may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in fig. 4, and are not described herein again.
It should be understood that, in the embodiment of the present invention, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In another embodiment of the invention, a computer-readable storage medium is provided. The computer readable storage medium may be a non-volatile computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program realizes the access amount control method disclosed by the embodiment of the invention when being executed by a processor.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, devices and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided by the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only a logical division, and there may be other divisions when the actual implementation is performed, or units having the same function may be grouped into one unit, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. An access amount control method, comprising:
acquiring a historical access request statistical data set stored locally, and acquiring an access active time interval according to the historical access request statistical data set;
if a current access request sent by a user side is received, acquiring a request type and request time corresponding to the current access request;
if the request time corresponding to the current access request is within the access active time interval, judging whether the current access request is a real-time reply request or a non-real-time reply request;
if the current access request is a real-time reply request, generating reply data corresponding to the current access request, and sending the reply data to a user side;
if the current access request is a non-real-time reply request, sending the current access request to a distributed log system for storage; and
and consuming the access request from the distributed log system by adopting a single thread mode, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
2. The access amount control method according to claim 1, wherein the obtaining a locally stored historical access request statistics data set, and obtaining an access active time interval according to the historical access request statistics data set, comprises:
acquiring historical access request statistical data subsets respectively corresponding to the historical access request statistical data sets every day;
acquiring historical access request distribution graphs respectively corresponding to the historical access request statistical data subsets according to access time points corresponding to the historical access requests in the historical access request statistical data subsets; the historical access request distribution curve takes a time axis as a horizontal axis and takes the number of the access requests as a vertical axis;
acquiring a part of each historical access request distribution curve graph, which is positioned above a preset access amount threshold curve, to obtain target distribution curve graphs corresponding to the historical access request distribution curve graphs respectively;
and acquiring time intervals corresponding to the target distribution curves, respectively averaging the starting time and the ending time of each time interval to obtain corresponding starting time average values and ending time average values, and taking the starting time average values as starting time points and the ending time average values as ending time points as access active time intervals.
3. The access amount control method according to claim 1, further comprising:
and if the request time corresponding to the current access request is not within the access active time interval, generating reply data corresponding to the current access request, and sending the reply data to the user side.
4. The access amount control method according to claim 1, wherein the consuming the access request from the distributed log system in the single thread mode, generating deferred reply data according to the access request obtained by the consuming, and sending the deferred reply data to the user side includes:
and if the time interval between the current single-thread processing time and the previous single-thread processing time is equal to a preset single-thread processing time threshold value, consuming the access request from the distributed log system, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
5. The access amount control method according to claim 1, wherein the consuming the access request from the distributed log system in the single thread mode, generating deferred reply data according to the access request obtained by the consuming, and sending the deferred reply data to the user side includes:
and if the current system time is not in the access active time interval, sequentially consuming the access requests from the distributed log system, correspondingly generating delayed reply data according to the access requests obtained by consumption, and sending the delayed reply data to the user side.
6. The access amount control method according to claim 1, wherein before consuming the access request from the distributed log system in the single thread mode, generating deferred reply data according to the consumed access request, and sending the deferred reply data to the user side, the method further comprises:
judging whether the distributed log system and the distributed log system are in the same internal local area network or not;
if the distributed log system is not in the same internal local area network as the distributed log system, acquiring an external network IP port of the distributed log system, and establishing connection with the distributed log system through the external network IP port;
and if the internal network IP port of the distributed log system is in the same internal local area network as the distributed log system, acquiring the internal network IP port of the distributed log system, and establishing connection with the distributed log system through the internal network IP port.
7. An access amount control apparatus, comprising:
the access active time interval acquisition unit is used for acquiring a locally stored historical access request statistical data set and acquiring an access active time interval according to the historical access request statistical data set;
the current access request analyzing unit is used for acquiring a request type and request time corresponding to a current access request if the current access request sent by a user side is received;
a current access request type determining unit, configured to determine whether the current access request is a real-time reply request or a non-real-time reply request if a request time corresponding to the current access request is within the access active time interval;
the first reply unit is used for generating reply data corresponding to the current access request and sending the reply data to the user side if the current access request is a real-time reply request;
the request forwarding unit is used for sending the current access request to a distributed log system for storage if the current access request is a non-real-time reply request; and
and the second reply unit is used for consuming the access request from the distributed log system in a single thread mode, correspondingly generating delayed reply data according to the access request obtained by consumption, and sending the delayed reply data to the user side.
8. The access amount control apparatus according to claim 7, wherein the access active time interval acquisition unit includes:
a history subset obtaining unit, configured to obtain history access request statistical data subsets corresponding to the history access request statistical data sets respectively every day;
a historical access request statistical data subset acquiring unit, configured to acquire historical access request distribution graphs corresponding to the historical access request statistical data subsets according to access time points corresponding to each historical access request in the historical access request statistical data subsets; the historical access request distribution curve takes a time axis as a horizontal axis and takes the number of the access requests as a vertical axis;
a target distribution curve graph obtaining unit, configured to obtain a portion, located above a preset access amount threshold curve, of each historical access request distribution curve graph, to obtain target distribution curve graphs corresponding to the historical access request distribution curve graphs, respectively;
and the time interval averaging unit is used for acquiring the time intervals corresponding to the target distribution graphs, averaging the starting time and the ending time of each time interval respectively to obtain corresponding starting time average values and ending time average values, and taking the time intervals with the starting time average values as the starting time points and the ending time average values as the ending time points as the access active time intervals.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the access amount control method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to execute the access amount control method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110124822.0A CN112764937B (en) | 2021-01-29 | 2021-01-29 | Access control method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110124822.0A CN112764937B (en) | 2021-01-29 | 2021-01-29 | Access control method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112764937A true CN112764937A (en) | 2021-05-07 |
| CN112764937B CN112764937B (en) | 2022-04-15 |
Family
ID=75706654
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110124822.0A Active CN112764937B (en) | 2021-01-29 | 2021-01-29 | Access control method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112764937B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201035774A (en) * | 2009-03-30 | 2010-10-01 | Mstar Semiconductor Inc | Method and apparatus for executing data access according to protocol handling |
| CN108073465A (en) * | 2017-12-29 | 2018-05-25 | 中国平安人寿保险股份有限公司 | Dynamic current limiting method, Nginx servers, storage medium and device |
| CN108683604A (en) * | 2018-04-03 | 2018-10-19 | 平安科技（深圳）有限公司 | concurrent access control method, terminal device and medium |
| CN109298990A (en) * | 2018-10-17 | 2019-02-01 | 平安科技（深圳）有限公司 | Log storing method, device, computer equipment and storage medium |
| CN109379395A (en) * | 2018-08-22 | 2019-02-22 | 中国平安人寿保险股份有限公司 | A kind of interface data buffer setting method and terminal device |
| CN109962855A (en) * | 2017-12-14 | 2019-07-02 | 深圳市融汇通金科技有限公司 | A kind of current-limiting method of WEB server, current-limiting apparatus and terminal device |
| CN110138669A (en) * | 2019-04-15 | 2019-08-16 | 中国平安人寿保险股份有限公司 | Interface access processing method, device, computer equipment and storage medium |
| CN110968745A (en) * | 2019-11-13 | 2020-04-07 | 泰康保险集团股份有限公司 | Data processing method and device, electronic equipment and computer readable medium |
| CN111131058A (en) * | 2019-11-25 | 2020-05-08 | 泰康保险集团股份有限公司 | Access point control method and device |
| CN111371772A (en) * | 2020-02-28 | 2020-07-03 | 深圳壹账通智能科技有限公司 | Intelligent gateway current limiting method and system based on redis and computer equipment |
| CN111479095A (en) * | 2020-06-24 | 2020-07-31 | 腾讯科技（深圳）有限公司 | Service processing control system, method and device |
| CN112035415A (en) * | 2020-08-28 | 2020-12-04 | 北京金山云网络技术有限公司 | Processing system, method, device and storage medium for user access data |
| US20200402058A1 (en) * | 2019-06-20 | 2020-12-24 | Coupang Corp. | Systems and methods for real-time processing of data streams |
-
2021
- 2021-01-29 CN CN202110124822.0A patent/CN112764937B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201035774A (en) * | 2009-03-30 | 2010-10-01 | Mstar Semiconductor Inc | Method and apparatus for executing data access according to protocol handling |
| CN109962855A (en) * | 2017-12-14 | 2019-07-02 | 深圳市融汇通金科技有限公司 | A kind of current-limiting method of WEB server, current-limiting apparatus and terminal device |
| CN108073465A (en) * | 2017-12-29 | 2018-05-25 | 中国平安人寿保险股份有限公司 | Dynamic current limiting method, Nginx servers, storage medium and device |
| CN108683604A (en) * | 2018-04-03 | 2018-10-19 | 平安科技（深圳）有限公司 | concurrent access control method, terminal device and medium |
| CN109379395A (en) * | 2018-08-22 | 2019-02-22 | 中国平安人寿保险股份有限公司 | A kind of interface data buffer setting method and terminal device |
| CN109298990A (en) * | 2018-10-17 | 2019-02-01 | 平安科技（深圳）有限公司 | Log storing method, device, computer equipment and storage medium |
| CN110138669A (en) * | 2019-04-15 | 2019-08-16 | 中国平安人寿保险股份有限公司 | Interface access processing method, device, computer equipment and storage medium |
| US20200402058A1 (en) * | 2019-06-20 | 2020-12-24 | Coupang Corp. | Systems and methods for real-time processing of data streams |
| CN110968745A (en) * | 2019-11-13 | 2020-04-07 | 泰康保险集团股份有限公司 | Data processing method and device, electronic equipment and computer readable medium |
| CN111131058A (en) * | 2019-11-25 | 2020-05-08 | 泰康保险集团股份有限公司 | Access point control method and device |
| CN111371772A (en) * | 2020-02-28 | 2020-07-03 | 深圳壹账通智能科技有限公司 | Intelligent gateway current limiting method and system based on redis and computer equipment |
| CN111479095A (en) * | 2020-06-24 | 2020-07-31 | 腾讯科技（深圳）有限公司 | Service processing control system, method and device |
| CN112035415A (en) * | 2020-08-28 | 2020-12-04 | 北京金山云网络技术有限公司 | Processing system, method, device and storage medium for user access data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112764937B (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP2021511588A (en) | Data query methods, devices and devices | |
| CN110740103A (en) | Service request processing method and device, computer equipment and storage medium | |
| WO2021104178A1 (en) | Dynamic message pushing method and system and vehicle diagnosis server | |
| US7698417B2 (en) | Optimized performance counter monitoring | |
| US9942272B2 (en) | Handling out of order events | |
| US20220327039A1 (en) | Creating statistical analyses of data for transmission to servers | |
| US20060277317A1 (en) | Providing flow control and moderation in a distributed message processing system | |
| US20160065241A1 (en) | Data transfer device, data transfer system, method for compressing and transferring data, and program | |
| CN112383585A (en) | Message processing system and method and electronic equipment | |
| CN110995618A (en) | Message processing method and device, electronic equipment and readable storage medium | |
| CN111240864A (en) | Asynchronous task processing method, device, equipment and computer readable storage medium | |
| CN110995566A (en) | Message data pushing method, system and device | |
| CN113010334A (en) | Request processing method, device and equipment | |
| CN110311963B (en) | Message pushing method and device, computer equipment and computer readable storage medium | |
| CN112764937B (en) | Access control method and device, computer equipment and storage medium | |
| CN103916463B (en) | Network access statistical analysis method and system | |
| CN116264592A (en) | Virtual desktop performance detection method, device, equipment and storage medium | |
| CN114138499B (en) | GPU resource utilization rate monitoring method and device, computer equipment and medium | |
| CN110659132A (en) | Request processing optimization method and computer-readable storage medium | |
| CN116226071A (en) | Data statistics method, device, equipment and storage medium | |
| CN107689979B (en) | method and equipment for processing download request | |
| CN108966160B (en) | Short message processing method and device and computer readable storage medium | |
| CN111737097B (en) | Performance test method and related device of stream processing system | |
| CN114461485A (en) | Resource loading monitoring method, device, equipment and storage medium | |
| CN117493048B (en) | Message dynamic processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |