CN110798470A - Proxy IP address management method and system - Google Patents
Proxy IP address management method and system Download PDFInfo
- Publication number
- CN110798470A CN110798470A CN201911054225.4A CN201911054225A CN110798470A CN 110798470 A CN110798470 A CN 110798470A CN 201911054225 A CN201911054225 A CN 201911054225A CN 110798470 A CN110798470 A CN 110798470A
- Authority
- CN
- China
- Prior art keywords
- pool
- proxy
- address
- sub
- addresses
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/50—Address allocation
- H04L61/5061—Pools of addresses
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/50—Address allocation
- H04L61/5053—Lease time; Renewal aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/59—Network arrangements, protocols or services for addressing or naming using proxies for addressing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0236—Filtering by address, protocol, port number or service, e.g. IP-address or URL
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Small-Scale Networks (AREA)
Abstract
The invention relates to a method and a system for managing an agent IP address, comprising the following steps: constructing a total pool, a garbage pool and at least one sub-pool, and storing a preset number of available proxy IP addresses into the total pool; each sub-pool corresponds to a service needing to be requested; according to the service attribute corresponding to each sub-pool, the total pool allocates agent IP addresses corresponding to the preset number to each sub-pool; the proxy IP address in the sub-pool is used for requesting a corresponding service; recording the times of each agent IP address in each sub-pool requesting corresponding services; when the frequency or the frequency of a service corresponding to a proxy IP address request in a certain sub-pool reaches a corresponding request threshold value, determining that the proxy IP address is forbidden by the corresponding service, and moving the forbidden proxy IP address to a garbage pool; and when the number of the agent IP addresses in the garbage pool reaches an update threshold value, switching each agent IP address in the garbage pool, correspondingly generating a new agent IP address, and storing each new agent IP address in the total pool. The invention can effectively detect the IP address state.
Description
Technical Field
The invention relates to the technical field of proxy I P addresses, in particular to a proxy IP address management method and a proxy IP address management system.
Background
With the development of network security, many websites increase the detection function for the number of times and frequency of IP (internet protocol address) requests, and once the IP is forbidden, the data collection work cannot be smoothly performed.
In the prior art, the treatment mode after the IP is forbidden is to restart the router, namely, the IP address is replaced, the proxy IP address can automatically redial, and a new IP is used for capturing. The method is only suitable for VPS (virtual private server) dial-up networking, and the measure is taken only after the IP is forbidden, so that the IP state cannot be automatically detected in time, and the possibility of the forbidden IP cannot be reduced.
Disclosure of Invention
The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art, and to provide a method and a system for proxy IP address management, which can automatically detect whether an IP is disabled, and can reduce the possibility of disabling the IP.
In order to achieve the above technical object, in one aspect, the present invention provides a method for managing an agent IP address, including:
constructing a total pool, a garbage pool and at least one sub-pool, and storing a preset number of available proxy IP addresses into the total pool; each sub-pool corresponds to a service needing to be requested;
according to the service attribute corresponding to each sub-pool, the total pool allocates agent IP addresses corresponding to the preset number to each sub-pool; the proxy IP address in the sub-pool is used for requesting a corresponding service;
recording the times of each agent IP address in each sub-pool requesting corresponding services;
when the frequency or the frequency of a service corresponding to a proxy IP address request in a certain sub-pool reaches a corresponding request threshold value, determining that the proxy IP address is forbidden by the corresponding service, and moving the forbidden proxy IP address to a garbage pool;
and when the number of the agent IP addresses in the garbage pool reaches an update threshold value, switching each agent IP address in the garbage pool, correspondingly generating a new agent IP address, and storing each new agent IP address in the total pool.
On the other hand, the proxy IP address management system provided by the present invention includes: the system comprises a main pool, a garbage pool and at least one sub-pool;
the total pool stores a preset number of available proxy IP addresses, and each sub-pool corresponds to a service to be requested;
the total pool is used for allocating agent IP addresses corresponding to the preset number to each sub-pool according to the service attribute corresponding to each sub-pool; the proxy IP address in the sub-pool is used for requesting a corresponding service;
the recording unit is used for recording the times of the service corresponding to each proxy IP address request in each sub-pool;
the judging unit is used for judging that the proxy IP address is forbidden by the corresponding service when the frequency or the frequency of the service corresponding to the proxy IP address request in a certain sub-pool reaches the corresponding request threshold value, and moving the forbidden proxy IP address to the garbage pool;
and the switching unit is used for switching each agent IP address in the garbage pool after the number of the agent IP addresses in the garbage pool reaches the updating threshold value, correspondingly generating new agent IP addresses, and storing each new agent IP address into the total pool.
In the invention, the total pool stores available proxy IP addresses and distributes the proxy IP addresses to the sub-pools; the sub-pool stores the proxy IP address of the service corresponding to the request; the agent IP address forbidden by the corresponding service is moved to a garbage pool; and the proxy IP address in the garbage pool can be updated and generated into a new proxy IP address through switching, and the new proxy IP address is stored to the total pool again. Therefore, proxy IP addresses always request corresponding services in each sub-pool, and the possibility that the services disable the IP is reduced. Because the times that each agent IP address in each sub-pool requests the corresponding service are recorded, the states that the agent IP address of each request service is forbidden and the like can be effectively and automatically detected.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating a method for proxy IP address management according to an exemplary embodiment of the present invention;
FIG. 2 is a block diagram of a proxy IP address management system in an exemplary embodiment of the present invention;
fig. 3 is a schematic diagram of a proxy IP address management display effect in an exemplary embodiment provided by the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, the present invention provides a proxy IP address management method, including:
101. constructing a total pool, a garbage pool and at least one sub-pool, and storing a preset number of available proxy IP addresses into the total pool; each sub-pool corresponds to a service needing to be requested;
102. according to the service attribute corresponding to each sub-pool, the total pool allocates agent IP addresses corresponding to the preset number to each sub-pool; the proxy IP address in the sub-pool is used for requesting a corresponding service;
103. recording the times of each agent IP address in each sub-pool requesting corresponding services;
104. when the frequency or the frequency of a service corresponding to a proxy IP address request in a certain sub-pool reaches a corresponding request threshold value, determining that the proxy IP address is forbidden by the corresponding service, and moving the forbidden proxy IP address to a garbage pool;
105. and when the number of the agent IP addresses in the garbage pool reaches an update threshold value, switching each agent IP address in the garbage pool, correspondingly generating a new agent IP address, and storing each new agent IP address in the total pool.
Furthermore, the set formed by the proxy IP addresses in each sub-pool is a subset of the set formed by the proxy IP addresses in the total pool, and the sets formed by the proxy IP addresses in each sub-pool can be intersected to generate an intersection;
the moving the forbidden proxy IP address to the garbage pool specifically includes:
judging whether the currently forbidden proxy IP address belongs to the proxy IP address in the intersection;
if not, moving the currently forbidden proxy IP address from the subpool to the garbage pool;
if yes, after the currently forbidden proxy IP address requests all the corresponding services of the sub-pool to which the currently forbidden proxy IP address belongs, the currently forbidden proxy IP address is stored in the garbage pool, and the currently forbidden proxy IP address is deleted in each sub-pool to which the currently forbidden proxy IP address belongs.
Further, the determining that the proxy IP address is disabled by the corresponding service further includes:
marking the currently forbidden proxy IP address in the total pool as an unallocated proxy IP address; the non-assignable proxy IP address will not be assigned to each sub-pool;
the switching of each proxy IP address in the garbage pool, the corresponding generation of new proxy IP addresses, and the storage of each new proxy IP address to the total pool specifically include:
switching each proxy IP address in the garbage pool, and correspondingly generating a new proxy IP address with different address contents;
and storing each new proxy IP address into the total pool, and replacing the corresponding proxy IP address before switching.
In the above technical solution, the switching each proxy IP address in the garbage pool further includes:
judging whether the switching is successful or not aiming at each proxy IP address in the garbage pool;
if yes, storing a new proxy IP address generated by the current proxy IP address into a total pool;
if not, continuing to switch the current proxy IP address, if no new proxy IP address is correspondingly generated in a preset time period, staying the current proxy IP address in the garbage pool, and sending a staying alarm.
Preferably, the storing each new agent IP address to the total pool further includes:
scanning the number of the proxy IP addresses in the total pool, the garbage pool and each sub-pool at a preset frequency;
when the number of the agent IP addresses in the total pool is smaller than the preset threshold value of the total pool, sending out a total pool increase alarm, and returning to switch each agent IP address in the garbage pool according to the total pool increase alarm and the scanned number of the agent IP addresses in the total pool until the number of the agent IP addresses in the total pool is the preset number;
when the number of the proxy IP addresses in a certain sub-pool is smaller than the preset threshold value of the corresponding sub-pool, sending out a corresponding sub-pool increase alarm, and according to the current sub-pool increase alarm and the number of the proxy IP addresses in the current sub-pool obtained by scanning, allocating proxy IP addresses to the current sub-pool by the total pool until the number of the proxy IP addresses in the current sub-pool obtained by scanning is the corresponding preset number;
when the number of the agent IP addresses in the garbage pool is larger than a preset threshold value of the garbage pool, sending a garbage pool reduction alarm;
the method further comprises the following steps:
and displaying the total pool, the garbage pool and each sub-pool.
As shown in fig. 2, the present invention provides a proxy IP address management system, including: a main tank 1, a garbage tank 3 and at least one sub-tank 2;
the total pool 1 stores a preset number of available proxy IP addresses, and each sub-pool 2 corresponds to a service to be requested;
the main pool 1 is used for allocating agent IP addresses corresponding to a preset number to each sub-pool 2 according to the service attribute corresponding to each sub-pool 2; the proxy IP address in the sub-pool 2 is used for requesting a corresponding service;
the recording unit 4 is used for recording the times of the service corresponding to each proxy IP address request in each sub-pool 2;
a determining unit 5, configured to determine that a certain proxy IP address in a certain sub-pool 2 is forbidden by a corresponding service when the number of times or frequency that the proxy IP address requests the corresponding service reaches a corresponding request threshold, and move the forbidden proxy IP address to the garbage pool 3;
and the switching unit 6 is used for switching each proxy IP address in the garbage pool 3 after the number of the proxy IP addresses in the garbage pool 3 reaches the update threshold value, correspondingly generating new proxy IP addresses, and storing each new proxy IP address into the total pool 1.
Further, the set formed by the proxy IP addresses in each sub-pool 2 is a subset of the set formed by the proxy IP addresses in the total pool 1, and the sets formed by the proxy IP addresses in each sub-pool 2 can be intersected to generate an intersection;
the determination unit 5 includes: a determination module 51 and a movement module 52;
the determining module 51 is configured to determine that a proxy IP address is disabled by a corresponding service when the number of times or frequency that the proxy IP address requests the corresponding service in a certain sub-pool 2 reaches a corresponding request threshold;
the moving module 52 is specifically configured to:
judging whether the currently forbidden proxy IP address belongs to the proxy IP address in the intersection;
if not, moving the currently forbidden proxy IP address from the sub-pool 2 to the garbage pool 3;
if yes, after the currently forbidden proxy IP address requests all the corresponding services of the sub-pool 2, the currently forbidden proxy IP address is stored in the garbage pool 3, and the currently forbidden proxy IP address is deleted in each sub-pool 2.
Further, the determination unit 5 further includes:
a marking module 53 for marking the currently disabled proxy IP address in the total pool as an unallocated proxy IP address; the non-assignable proxy IP address will not be assigned to each sub-pool 2;
the switching unit 6 is specifically configured to:
switching is carried out aiming at each agent IP address in the garbage pool 3, and a new agent IP address with different address contents is correspondingly generated;
each new proxy IP address is stored in the pool 1 and replaces the corresponding proxy IP address before the handover.
In the above technical solution, the switching unit 6 is further configured to:
judging whether the switching is successful or not aiming at each agent IP address in the garbage pool 3;
if yes, storing a new proxy IP address generated by the current proxy IP address into a total pool 1;
if not, continuing to switch the current proxy IP address, if no new proxy IP address is correspondingly generated in a preset time period, staying the current proxy IP address in the garbage pool 3, and sending a staying alarm.
Preferably, the system further comprises:
a scanning unit 7, configured to scan the number of proxy IP addresses in the total pool 1, the junk pool 3, and each sub-pool 2 at a predetermined frequency;
the alarm unit 8 is used for sending out a total pool increase alarm when the number of the proxy IP addresses in the total pool 1 is smaller than a preset total pool threshold value, and returning to the switching unit 4 according to the total pool increase alarm and the scanned number of the proxy IP addresses in the total pool 1 until the scanning unit 7 obtains that the number of the proxy IP addresses in the total pool is a preset number;
when the number of the proxy IP addresses in a certain sub-pool 2 is smaller than the preset threshold value of the corresponding sub-pool, sending out a corresponding sub-pool increase alarm, and according to the current sub-pool 2 increase alarm and the number of the proxy IP addresses in the current sub-pool 2 obtained by scanning, enabling the total pool 1 to allocate the proxy IP addresses to the current sub-pool 2 until the scanning unit 7 obtains that the number of the proxy IP addresses in the current sub-pool 2 is the corresponding preset number;
when the number of the agent IP addresses in the garbage pool 3 is larger than a preset threshold value of the garbage pool, sending a garbage pool reduction alarm;
the system, still include:
and the display unit 9 is used for displaying the total pool 1, the garbage pool 3 and each sub-pool 2.
In the invention, a total pool 1 stores available proxy IP addresses and distributes the proxy IP addresses to a sub-pool; the sub-pool 2 stores the proxy IP address of the service corresponding to the request; the agent IP address forbidden by the corresponding service is moved to a garbage pool 3; and the proxy IP address in the garbage pool 3 can be updated to generate a new proxy IP address through switching, and the new proxy IP address is stored to the total pool 1 again. Therefore, each sub-pool 2 always has an agent IP address to request corresponding service, thereby reducing the possibility that the service disables IP. Since the number of times that the proxy IP addresses in each sub-pool 2 request the corresponding service is recorded, the case where the proxy IP address of each requested service is disabled can be automatically detected.
In the invention, each service corresponds to one sub-pool 2, each agent IP address in the sub-pool 2 is available, and when a certain agent IP address in the sub-pool 2 is forbidden, the service can be moved to a garbage pool. Thus, the provision of sub-pool 2 solves the problem of service disablement where a single IP is requested. The invention can find the forbidden proxy IP address in time and then efficiently replace the new proxy IP address. The total pool 1, the garbage pool 3 and each sub-pool 2 are displayed, the change of each agent IP address can be observed, the total pool 1, the garbage pool 3 and each sub-pool 2 are convenient to operate, and inconvenience caused by non-visualization is avoided.
The above technical solutions of the embodiments of the present invention are described in detail below with reference to application examples:
as shown in fig. 3, the method for managing an agent IP address according to this embodiment includes:
step 1, constructing a total pool, a garbage pool and three sub-pools, and storing a preset number of available proxy IP addresses into the total pool; each sub-pool corresponds to a service to be requested.
In fig. 3, service 1 corresponds to a first sub-pool, service 2 corresponds to a second sub-pool, and service 3 corresponds to a third sub-pool. The 20 available proxy IP addresses stored in the total pool: IP1, IP2 … … IP 20. Each service corresponds to a website in this embodiment, and the proxy IP address in each sub-pool is used to send a request to the corresponding website.
The proxy IP address allocated to the first sub-pool by the total pool is: IP1, IP2, and IP 3; the proxy IP address allocated by the total pool to the second sub-pool is: IP4, IP5, and IP 6; the proxy IP address allocated to the third sub-pool by the total pool is: IP7, IP8, and IP 9. After the agent IP addresses are distributed to all the sub-pools by the main pool, the distributed agent IP addresses still exist in the main pool, and only after the garbage pool is switched, the agent IP addresses in the main pool are possible to change.
The set of proxy IP addresses in the first through third sub-pools is a subset of the set of proxy IP addresses in the total pool. The sets formed by the proxy IP addresses in each sub-pool can be intersected to generate intersection; however, for simplicity of example, in this embodiment, the sets of proxy IP addresses in the first to third sub-pools do not intersect.
And 3, recording the times of the service corresponding to each proxy IP address request in each sub-pool.
In the present embodiment, the IP pools (the total pool, the sub-pool, and the garbage pool) are stored in the form of hash (hash algorithm), together with the IP information. For example: { "Total pool of available IPs" { IP1: count 1; IP2: count2} }. The key is the name of the IP pool, and the value is also a hash structure, wherein the key is the IP address, and the value is the number of times that the proxy IP address requests the corresponding service. Thus, the times of requesting corresponding services by each proxy IP address in each sub-pool can be recorded.
And 4, when the frequency or the frequency of the service corresponding to the agent IP address request in a certain sub-pool reaches a corresponding request threshold value, judging that the agent IP address is forbidden by the corresponding service, and moving the forbidden agent IP address to a garbage pool.
The disabling mode of each service to the IP is different, some services are requested by limiting the times, and some services are requested by limiting the frequency. The request thresholds set by different services are different, some are the total number of requests, and some are the frequency in a certain time period. Since the frequency is related to the number of times in the cycle, in this embodiment, the number of times that each proxy IP address in each sub-pool requests the corresponding service is recorded, and the frequency can also be used in the service for limiting the request frequency. For example, a web site is disabled for IP for 3 visits per day for a week. In this embodiment, the number of requests for the proxy IP address in each sub-pool is grasped, and then, according to the recorded time, the data corresponding to the frequency can be obtained, thereby determining whether the frequency reaches the corresponding request threshold.
Step 4.1, when the times or frequency of a service corresponding to a certain proxy IP address request in a certain sub-pool reaches a corresponding request threshold value, determining that the proxy IP address is forbidden by the corresponding service;
step 4.2, judging whether the currently forbidden proxy IP address belongs to the proxy IP address in the intersection;
4.3, if not, moving the currently forbidden proxy IP address from the subpool to the garbage pool; for the embodiment in fig. 3, the proxy IP addresses of the respective sub-pools do not intersect, so any proxy IP address in the first sub-pool to the third sub-pool is directly moved to the garbage pool after being disabled.
And 4.4, if so, storing the currently forbidden proxy IP address to a garbage pool after the currently forbidden proxy IP address requests all the corresponding services of the sub-pool to which the currently forbidden proxy IP address belongs, and deleting the currently forbidden proxy IP address in each sub-pool to which the currently forbidden proxy IP address belongs.
For the case described in step 4.4, assume that the total pool allocates IP20 to the first, second and third sub-pools simultaneously, while IP20 is disabled by service 1 at this time:
at this time, the IP20 is determined to belong to the proxy IP address in the intersection, and it is necessary to wait until the IP20 requests both the service 2 and the service 3, and then store the IP20 in the garbage pool, and delete the IP20 in the first sub-pool, the second sub-pool, and the third sub-pool. I.e. mobility agent IP address, the agent IP address cannot be in the request state. Therefore, the agent IP address is prevented from being invalid after being moved to the garbage pool, and the invalid agent IP address is just in a state of requesting other services.
Step 4.5, marking the currently forbidden proxy IP address in the total pool as an unallocated proxy IP address; the non-assignable proxy IP address will not be assigned to each sub-pool;
for example, IP20 is marked as an unallocated proxy IP address when disabled by service 1, and even while waiting for IP20 to complete requesting service 2 and service 3, IP20 in the overall pool is not reallocated. Thus, the logical and rational operation of the present embodiment is ensured.
And 5, when the number of the proxy IP addresses in the garbage pool reaches an update threshold value, switching each proxy IP address in the garbage pool, correspondingly generating a new proxy IP address, and storing each new proxy IP address in the total pool.
Step 5.1, switching is carried out aiming at each proxy IP address in the garbage pool, and a new proxy IP address with different address contents is correspondingly generated;
as shown in fig. 3, clicking "switch IP service" directly performs step 5.1. For example, IP20 has been placed in the garbage pool at this time, and clicking "switch IP service" updates IP20 to IP 200.
Step 5.2, storing each new agent IP address into a total pool, and replacing the corresponding agent IP address before switching;
for example, after IP20 is updated to IP200, it is moved from the garbage pool to the general pool to replace IP20 that was marked as unallocated, by which point IP20 disappears completely in each IP pool.
Step 5.3, judging whether the switching is successful or not aiming at each proxy IP address in the garbage pool;
if yes, storing a new proxy IP address generated by the current proxy IP address into a total pool;
if not, continuing to switch the current proxy IP address, if no new proxy IP address is correspondingly generated in a preset time period, staying the current proxy IP address in the garbage pool, and sending a staying alarm.
In this embodiment, a service is independently provided for switching the proxy IP address. And when the proxy IP address needs to be switched, calling the service interface. At this time, another service is needed to detect whether the proxy IP address is successfully switched, the principle is that each new proxy IP address requests IP138 (a website for IP address detection), if the IP address changes, the switching is successful, and the IP address is stored in the total pool; otherwise, continuing to call the service of switching the IP; and setting overtime time, if the agent IP address cannot be successfully switched within 5 minutes, sending an alarm mail to prompt that the IP service is switched to be in a problem, and then maintaining the IP service at the first time.
Step 6, scanning the number of the proxy IP addresses in the total pool, the garbage pool and each sub-pool at a preset frequency;
7.1, when the number of the proxy IP addresses in the total pool is smaller than a preset threshold value of the total pool, sending a total pool increase alarm, and returning to switch each proxy IP address in the garbage pool according to the total pool increase alarm and the scanned number of the proxy IP addresses in the total pool until the scanned number of the proxy IP addresses in the total pool is a preset number;
7.2, when the number of the proxy IP addresses in a certain sub-pool is smaller than the preset threshold value of the corresponding sub-pool, sending out a corresponding sub-pool increase alarm, and according to the current sub-pool increase alarm and the number of the proxy IP addresses in the current sub-pool obtained by scanning, allocating proxy IP addresses to the current sub-pool by the total pool until the number of the proxy IP addresses in the current sub-pool obtained by scanning is the corresponding preset number;
and 7.3, sending a garbage pool reduction alarm when the number of the proxy IP addresses in the garbage pool is greater than a preset threshold value of the garbage pool.
In this embodiment, Prometheus can be used to monitor the number of proxy IP addresses in each sub-pool and the number of proxy IP addresses in the garbage pool, and alert mail can be sent using alert manager. When the number of the proxy IP addresses in one sub-pool is continuously less than the preset threshold value of the corresponding sub-pool, sending an email alarm (Prometheus scans the number of the IP addresses in each IP pool at a custom time interval, and if the number of the IP addresses of the continuous scanning times is less than the corresponding threshold value, an alert manager is triggered to send an alarm email), and repairing can be carried out in the first time. When the number of IP in the garbage pool is continuously higher than the preset threshold value of the garbage pool, sending a mail alarm, and then repairing the mail in the first time. The reduced alarms for the garbage pool may occur because the switching service has a problem, resulting in unsuccessful switching of the proxy IP address in the garbage pool, and thus accumulated in the garbage pool.
The embodiment of the present invention provides a proxy IP address management system, which can implement the method embodiment provided above, and for specific function implementation, reference is made to the description in the method embodiment, which is not described herein again.
The invention aims at the problems of the existing social network platform and the advertiser system, effectively solves the problems of long conversion path, high information search cost, high loss rate and the like of a user, and when the user browses advertisements and is interested in commodities, clicks the corresponding pre-purchase button, directly adds the commodities into a shopping cart or jumps to a shopping page, thereby directly ordering and purchasing the commodities and greatly improving the commodity volume. In addition, the social network background can also recycle related data effects, such as user ordering situation, total volume of deals effect conversion, and the like. Therefore, the social network platform can monitor the conversion effect of each layer of the conversion path in time, and provide data base support for subsequent related strategy optimization and the like.
It should be understood that the specific order or hierarchy of steps in the processes disclosed is an example of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not intended to be limited to the specific order or hierarchy presented.
In the foregoing detailed description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments of the subject matter require more features than are expressly recited in each claim. Rather, as the following claims reflect, invention lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. To those skilled in the art; various modifications to these embodiments will be readily apparent, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the aforementioned embodiments, but one of ordinary skill in the art may recognize that many further combinations and permutations of various embodiments are possible. Accordingly, the embodiments described herein are intended to embrace all such alterations, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional word in a claim. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or".
Those of skill in the art will further appreciate that the various illustrative logical blocks, units, and steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, elements, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements of the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.
The various illustrative logical blocks, or elements, described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor, an Application Specific Integrated Circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. For example, a storage medium may be coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC, which may be located in a user terminal. In the alternative, the processor and the storage medium may reside in different components in a user terminal.
In one or more exemplary designs, the functions described above in connection with the embodiments of the invention may be implemented in hardware, software, firmware, or any combination of the three. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media that facilitate transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a general purpose or special purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store program code in the form of instructions or data structures and which can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Additionally, any connection is properly termed a computer-readable medium, and, thus, is included if the software is transmitted from a website, server, or other remote source via a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wirelessly, e.g., infrared, radio, and microwave. Such discs (disk) and disks (disc) include compact disks, laser disks, optical disks, DVDs, floppy disks and blu-ray disks where disks usually reproduce data magnetically, while disks usually reproduce data optically with lasers. Combinations of the above may also be included in the computer-readable medium.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for proxy IP address management, the method comprising:
constructing a total pool, a garbage pool and at least one sub-pool, and storing a preset number of available proxy IP addresses into the total pool; each sub-pool corresponds to a service needing to be requested;
according to the service attribute corresponding to each sub-pool, the total pool allocates agent IP addresses corresponding to the preset number to each sub-pool; the proxy IP address in the sub-pool is used for requesting a corresponding service;
recording the times of each agent IP address in each sub-pool requesting corresponding services;
when the frequency or the frequency of a service corresponding to a proxy IP address request in a certain sub-pool reaches a corresponding request threshold value, determining that the proxy IP address is forbidden by the corresponding service, and moving the forbidden proxy IP address to a garbage pool;
and when the number of the agent IP addresses in the garbage pool reaches an update threshold value, switching each agent IP address in the garbage pool, correspondingly generating a new agent IP address, and storing each new agent IP address in the total pool.
2. The method of claim 1, wherein the set of proxy IP addresses in each sub-pool is a subset of the set of proxy IP addresses in the total pool, and the sets of proxy IP addresses in the sub-pools can intersect to form an intersection;
the moving the forbidden proxy IP address to the garbage pool specifically includes:
judging whether the currently forbidden proxy IP address belongs to the proxy IP address in the intersection;
if not, moving the currently forbidden proxy IP address from the subpool to the garbage pool;
if yes, after the currently forbidden proxy IP address requests all the corresponding services of the sub-pool to which the currently forbidden proxy IP address belongs, the currently forbidden proxy IP address is stored in the garbage pool, and the currently forbidden proxy IP address is deleted in each sub-pool to which the currently forbidden proxy IP address belongs.
3. The method of claim 2, wherein the determining that the proxy IP address is disabled by the corresponding service further comprises:
marking the currently forbidden proxy IP address in the total pool as an unallocated proxy IP address; the non-assignable proxy IP address will not be assigned to each sub-pool;
the switching of each proxy IP address in the garbage pool, the corresponding generation of new proxy IP addresses, and the storage of each new proxy IP address to the total pool specifically include:
switching each proxy IP address in the garbage pool, and correspondingly generating a new proxy IP address with different address contents;
and storing each new proxy IP address into the total pool, and replacing the corresponding proxy IP address before switching.
4. The method according to claim 1, wherein the switching each proxy IP address in the garbage pool further comprises:
judging whether the switching is successful or not aiming at each proxy IP address in the garbage pool;
if yes, storing a new proxy IP address generated by the current proxy IP address into a total pool;
if not, continuing to switch the current proxy IP address, if no new proxy IP address is correspondingly generated in a preset time period, staying the current proxy IP address in the garbage pool, and sending a staying alarm.
5. The method of claim 1, wherein storing each new proxy IP address to a total pool, further comprises:
scanning the number of the proxy IP addresses in the total pool, the garbage pool and each sub-pool at a preset frequency;
when the number of the agent IP addresses in the total pool is smaller than the preset threshold value of the total pool, sending out a total pool increase alarm, and returning to switch each agent IP address in the garbage pool according to the total pool increase alarm and the scanned number of the agent IP addresses in the total pool until the number of the agent IP addresses in the total pool is the preset number;
when the number of the proxy IP addresses in a certain sub-pool is smaller than the preset threshold value of the corresponding sub-pool, sending out a corresponding sub-pool increase alarm, and according to the current sub-pool increase alarm and the number of the proxy IP addresses in the current sub-pool obtained by scanning, allocating proxy IP addresses to the current sub-pool by the total pool until the number of the proxy IP addresses in the current sub-pool obtained by scanning is the corresponding preset number;
when the number of the agent IP addresses in the garbage pool is larger than a preset threshold value of the garbage pool, sending a garbage pool reduction alarm;
the method further comprises the following steps:
and displaying the total pool, the garbage pool and each sub-pool.
6. A proxy IP address management system, the system comprising: the system comprises a main pool, a garbage pool and at least one sub-pool;
the total pool stores a preset number of available proxy IP addresses, and each sub-pool corresponds to a service to be requested;
the total pool is used for allocating agent IP addresses corresponding to the preset number to each sub-pool according to the service attribute corresponding to each sub-pool; the proxy IP address in the sub-pool is used for requesting a corresponding service;
the recording unit is used for recording the times of the service corresponding to each proxy IP address request in each sub-pool;
the judging unit is used for judging that the proxy IP address is forbidden by the corresponding service when the frequency or the frequency of the service corresponding to the proxy IP address request in a certain sub-pool reaches the corresponding request threshold value, and moving the forbidden proxy IP address to the garbage pool;
and the switching unit is used for switching each agent IP address in the garbage pool after the number of the agent IP addresses in the garbage pool reaches the updating threshold value, correspondingly generating new agent IP addresses, and storing each new agent IP address into the total pool.
7. The proxy IP address management system of claim 6 wherein the set of proxy IP addresses in each sub-pool is a subset of the set of proxy IP addresses in the total pool, the sets of proxy IP addresses in the sub-pools being intersected to produce an intersection;
the determination unit includes: a judging module and a moving module;
the judging module is used for judging that the proxy IP address is forbidden by the corresponding service when the frequency or the frequency of the service corresponding to the proxy IP address request in a certain sub-pool reaches the threshold value of the corresponding request;
the moving module is specifically configured to:
judging whether the currently forbidden proxy IP address belongs to the proxy IP address in the intersection;
if not, moving the currently forbidden proxy IP address from the subpool to the garbage pool;
if yes, after the currently forbidden proxy IP address requests all the corresponding services of the sub-pool to which the currently forbidden proxy IP address belongs, the currently forbidden proxy IP address is stored in the garbage pool, and the currently forbidden proxy IP address is deleted in each sub-pool to which the currently forbidden proxy IP address belongs.
8. The proxy IP address management system according to claim 7, wherein the determination unit further includes:
the marking module marks the currently forbidden proxy IP address in the total pool as an unallocated proxy IP address; the non-assignable proxy IP address will not be assigned to each sub-pool;
the switching unit is specifically configured to:
switching each proxy IP address in the garbage pool, and correspondingly generating a new proxy IP address with different address contents;
and storing each new proxy IP address into the total pool, and replacing the corresponding proxy IP address before switching.
9. The proxy IP address management system of claim 6, wherein the switching unit is further configured to:
judging whether the switching is successful or not aiming at each proxy IP address in the garbage pool;
if yes, storing a new proxy IP address generated by the current proxy IP address into a total pool;
if not, continuing to switch the current proxy IP address, if no new proxy IP address is correspondingly generated in a preset time period, staying the current proxy IP address in the garbage pool, and sending a staying alarm.
10. The proxy IP address management system according to claim 6, further comprising:
the scanning unit is used for scanning the number of the proxy IP addresses in the total pool, the garbage pool and each sub-pool at a preset frequency;
the alarm unit is used for sending out a total pool increase alarm when the number of the proxy IP addresses in the total pool is smaller than a preset total pool threshold value, and returning to the switching unit according to the total pool increase alarm and the number of the proxy IP addresses in the total pool obtained by scanning until the number of the proxy IP addresses in the total pool obtained by the scanning unit is the preset number;
when the number of the proxy IP addresses in a certain sub-pool is smaller than the preset threshold value of the corresponding sub-pool, sending out a corresponding sub-pool increase alarm, and enabling the total pool to allocate the proxy IP addresses to the current sub-pool according to the current sub-pool increase alarm and the number of the proxy IP addresses in the current sub-pool obtained by scanning until the scanning unit obtains the number of the proxy IP addresses in the current sub-pool as the corresponding preset number;
when the number of the agent IP addresses in the garbage pool is larger than a preset threshold value of the garbage pool, sending a garbage pool reduction alarm;
the system, still include:
and the display unit is used for displaying the total pool, the garbage pool and each sub-pool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911054225.4A CN110798470B (en) | 2019-10-31 | 2019-10-31 | Proxy IP address management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911054225.4A CN110798470B (en) | 2019-10-31 | 2019-10-31 | Proxy IP address management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110798470A true CN110798470A (en) | 2020-02-14 |
| CN110798470B CN110798470B (en) | 2022-08-23 |
Family
ID=69442418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911054225.4A Active CN110798470B (en) | 2019-10-31 | 2019-10-31 | Proxy IP address management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110798470B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117278526A (en) * | 2023-11-21 | 2023-12-22 | 厦门市唯云网络科技有限公司 | Proxy IP address scheduling method and device based on artificial neural network |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103533097A (en) * | 2013-10-10 | 2014-01-22 | 北京京东尚科信息技术有限公司 | Web crawler downloading and analyzing method and device |
| CN105825399A (en) * | 2016-03-15 | 2016-08-03 | 焦点科技股份有限公司 | Internet based B2B e-commerce information collecting method |
| CN106254579A (en) * | 2016-09-30 | 2016-12-21 | 携程计算机技术（上海）有限公司 | A kind of dynamic IP agent pool and building and management method |
| US20170019874A1 (en) * | 2014-04-03 | 2017-01-19 | Huawei Technologies Co., Ltd. | Ip address assignment apparatus, system, and method |
| CN108345642A (en) * | 2018-01-12 | 2018-07-31 | 深圳壹账通智能科技有限公司 | Method, storage medium and the server of website data are crawled using Agent IP |
| CN110149419A (en) * | 2019-05-23 | 2019-08-20 | 上海睿翎法律咨询服务有限公司 | The efficient crawler method of IP-based |
| CN110249595A (en) * | 2016-11-09 | 2019-09-17 | 斯高奇代理有限责任公司 | System and method for communication request to be routed to dedicated proxies |
| WO2019200784A1 (en) * | 2018-04-18 | 2019-10-24 | 平安科技（深圳）有限公司 | Method for web crawler, terminal, and storage medium |
-
2019
- 2019-10-31 CN CN201911054225.4A patent/CN110798470B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103533097A (en) * | 2013-10-10 | 2014-01-22 | 北京京东尚科信息技术有限公司 | Web crawler downloading and analyzing method and device |
| US20170019874A1 (en) * | 2014-04-03 | 2017-01-19 | Huawei Technologies Co., Ltd. | Ip address assignment apparatus, system, and method |
| CN105825399A (en) * | 2016-03-15 | 2016-08-03 | 焦点科技股份有限公司 | Internet based B2B e-commerce information collecting method |
| CN106254579A (en) * | 2016-09-30 | 2016-12-21 | 携程计算机技术（上海）有限公司 | A kind of dynamic IP agent pool and building and management method |
| CN110249595A (en) * | 2016-11-09 | 2019-09-17 | 斯高奇代理有限责任公司 | System and method for communication request to be routed to dedicated proxies |
| CN108345642A (en) * | 2018-01-12 | 2018-07-31 | 深圳壹账通智能科技有限公司 | Method, storage medium and the server of website data are crawled using Agent IP |
| WO2019200784A1 (en) * | 2018-04-18 | 2019-10-24 | 平安科技（深圳）有限公司 | Method for web crawler, terminal, and storage medium |
| CN110149419A (en) * | 2019-05-23 | 2019-08-20 | 上海睿翎法律咨询服务有限公司 | The efficient crawler method of IP-based |
Non-Patent Citations (1)
| Title |
|---|
| IAMMAPPING: "基于Redis的代理ip池设计", 《HTTPS://JUEJIN.CN/POST/6844903461264293895》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117278526A (en) * | 2023-11-21 | 2023-12-22 | 厦门市唯云网络科技有限公司 | Proxy IP address scheduling method and device based on artificial neural network |
| CN117278526B (en) * | 2023-11-21 | 2024-03-19 | 厦门市唯云网络科技有限公司 | Proxy IP address scheduling method and device based on artificial neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110798470B (en) | 2022-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2009347686B2 (en) | Method and device for flow counting and automatically calculating internet connection fee | |
| US8874091B2 (en) | Automatic device capabilities change notification | |
| CN102782670B (en) | Memory cache data center | |
| CN110677405B (en) | Data processing method and device, electronic equipment and storage medium | |
| US11770438B2 (en) | Network communications | |
| CN112087644B (en) | Pull stream request processing method, device and system, electronic equipment and storage medium | |
| KR20060084040A (en) | Apparatus and method for dynamic qos management | |
| WO2018214793A1 (en) | Method and device for providing sharing-related feedback or excitation information associated with wireless access point | |
| BR112012003248B1 (en) | method and server for instant messaging | |
| TW201340011A (en) | Information recommendation method and information recommendation apparatus | |
| US20130031197A1 (en) | Internet cache subscription for wireless mobile users | |
| CN101562578A (en) | Method and device for notifying user data subscription | |
| CN110798470A (en) | Proxy IP address management method and system | |
| CN101771671A (en) | Method and device for interactive processing between content identity management servers | |
| US20130232208A1 (en) | Method and device for updating messages | |
| JP2007188239A (en) | Document management system | |
| JP2013239992A (en) | Telephone number list management server, telephone number list management program, and method for managing telephone number list | |
| CN111767481B (en) | Access processing method, device, equipment and storage medium | |
| CN115297085B (en) | Domain name resolution updating method and device, electronic equipment and storage medium | |
| CN112118487B (en) | Video playing control method, device, equipment and medium | |
| CN1543612A (en) | Method and system for registering and searching user position information | |
| CN110351719A (en) | A kind of wireless network management method, system and electronic equipment and storage medium | |
| JP2009277227A (en) | Filtering method for advertisement blocking function and system | |
| JP2021182425A (en) | Information management device and information management method | |
| WO2010054554A1 (en) | Method, system and server for centralized management of network service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |