CN113905092A - Method, device, terminal and storage medium for determining reusable agent queue - Google Patents
Method, device, terminal and storage medium for determining reusable agent queue Download PDFInfo
- Publication number
- CN113905092A CN113905092A CN202111142651.0A CN202111142651A CN113905092A CN 113905092 A CN113905092 A CN 113905092A CN 202111142651 A CN202111142651 A CN 202111142651A CN 113905092 A CN113905092 A CN 113905092A
- Authority
- CN
- China
- Prior art keywords
- agent
- environment
- current
- target website
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 12
- 230000004044 response Effects 0.000 claims description 54
- 238000004590 computer program Methods 0.000 claims description 7
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 6
- 238000012423 maintenance Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000001681 protective effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000004883 computer application Methods 0.000 description 1
- 230000008260 defense mechanism Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Landscapes
- Computer And Data Communications (AREA)
Abstract
The invention discloses a method for determining a reusable agent queue, and discloses a device, a terminal and a storage medium corresponding to the method. The method for determining the reusable agent queue determines whether the agent environment is suitable for the target website or not by distributing the agent environment to the target website which the client wants to access, and according to the access result of the agent environment, if so, the agent environment is stored in the reusable agent queue corresponding to the target website, so that other clients can conveniently access the target website, and the processing efficiency is improved.
Description
Technical Field
The invention relates to the field of computer application design, in particular to a method, a device, a terminal and a storage medium for determining a reusable agent queue.
Background
When any network is built, protective measures are generally adopted, and the most common protective measure among the protective measures is to limit the access frequency of each IP, that is, the information acquired by one IP on a target website in unit time is very limited.
However, when collecting network data, the information acquisition efficiency is not high due to the protection mechanism of the target network, and it is difficult to collect the information on the target website comprehensively.
In order to solve the problems occurring in the process of collecting network data, the prior art generally adopts an IP proxy pool method or a random tunnel proxy method. However, the existing scheme has a series of problems of low management efficiency, unsatisfactory connection speed, unsafe threads and the like.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the invention provides a method for determining a reusable agent queue, which can provide a scheme with high concurrency controllability and rapid processing.
The invention also provides equipment, a terminal and a storage medium for determining the reusable agent queue.
According to a first aspect of the present invention, a method for determining a reusable agent queue is provided, which includes the following steps:
acquiring an access request from a client;
determining a current agent environment applicable to the access request;
sending the access request to a target website based on the current proxy environment, and judging whether the current proxy environment is suitable for the target website according to a response message of the target website;
and if the current agent environment is suitable for the target website, storing the current agent environment to a reusable agent environment queue corresponding to the target website.
The method for determining the reusable agent queue according to the embodiment of the invention at least has the following beneficial effects: after the agent environment is distributed to the client and the request is accessed, a corresponding reusable agent environment queue can be established for each target website according to the response message of the target website, so that the client accessing the target website subsequently can be ensured, the processed agent environment which can be directly used can be selected from the reusable agent queue, and the access efficiency is improved.
According to some embodiments of the invention, the step of determining a current proxy context applicable to the access request comprises:
determining a target website to be accessed by the access request, and acquiring a reusable agent environment queue corresponding to the target website;
if available agent environments exist in the reusable agent environment queue, selecting the agent environment from the reusable agent environment queue as the current agent environment;
and if the reusable agent environment queue does not have available agent environments, selecting agents which are not in a blacklist or a grey list from the available agent queue as current agent environments, wherein the blacklist records agents which are not suitable for the target website, and the grey list records agents which are in use.
According to some embodiments of the invention, the step of selecting an agent from the queue of available agents that is not in the blacklist or the grey list as the current agent context comprises:
selecting agents from the queue of available agents that are not on the blacklist or the grey list;
and combining the agent with the website information of the target website, and taking a combination result as a current agent environment corresponding to the target website.
According to some embodiments of the invention, the step of determining the available agent queues comprises:
acquiring a source agent from a third party agent and placing the source agent into an agent pool;
determining a response speed level of each of the source agents in an agent pool;
and using the response speed level of each source agent as a storage priority to store the source agents into an available agent queue at regular time.
According to some embodiments of the present invention, the step of determining whether the current proxy environment is applicable to the target website according to the response message of the target website includes:
if the response status code in the response message is an environment failure status code, determining that the current agent environment is not applicable to the target website;
if the connection with the target website cannot be established through the current agent environment, judging that the current agent environment is not suitable for the target website;
if the connection with the target website can be established through the current agent environment, but the response of receiving the target website is overtime, judging that the current agent environment is not suitable for the target website;
if the response status code in the response message is the server abnormal status code, judging that the current agent environment is not suitable for the target website;
and if the other conditions exist, judging that the current agent environment is suitable for the target website.
According to some embodiments of the present invention, after determining whether the current proxy environment is applicable to the target website according to the response message of the target website, the method further includes:
and if the current agent environment is not suitable for the target website, detecting the self state of the current agent environment and correspondingly processing the current agent environment according to the detection result.
According to some embodiments of the present invention, the step of detecting the self-state of the current agent environment and correspondingly processing the current agent environment according to the detection result if the current agent environment is not applicable to the target website specifically includes:
accessing a public network server through the current proxy environment;
if the current agent environment can access the public network server, the current agent environment is not processed;
if the current agent environment can not establish network connection with a public network server, deleting the current agent environment from an available agent queue and writing the current agent environment into a blacklist;
if the current agent environment successfully establishes network connection with a public network server, but the response of a receiving server is overtime, deleting the current agent environment from an available agent queue and writing the current agent environment into a blacklist;
and if the response status code has an abnormal status code after the network connection between the current agent environment and the public network server is established, deleting the current agent environment from the available agent queue and writing the current agent environment into a blacklist.
The apparatus for determining a reusable agent queue according to the second aspect of the present application is characterized by comprising:
the access request acquisition module can acquire an access request from a client;
the agent environment distribution module can determine the current agent environment applicable to the access request;
the response processing module can send the access request to a target website based on the current proxy environment and judge whether the current proxy environment is suitable for the target website according to the response message of the target website;
and the queue maintenance module is used for storing the current agent environment to a reusable agent environment queue corresponding to the target website if the current agent environment is suitable for the target website.
The device for determining the reusable agent queue according to the embodiment of the invention at least has the following beneficial effects: the device can determine whether the current agent environment is suitable for the target website or not based on the website access result after the agent environment is distributed to the client, and then the agent environment suitable for the target website is stored in the reusable agent queue by using the queue maintenance module, so that the access efficiency of the subsequent process is improved.
Further, the agent environment allocation module further includes:
the queue selection element can determine a target website to be accessed by the access request and acquire a reusable agent environment queue corresponding to the target website;
a first agent environment selecting element, which selects an agent environment from the reusable agent environment queue as a current agent environment if an available agent environment exists in the reusable agent environment queue;
and a second agent environment selecting element, configured to select, as a current agent environment, an agent that is not on a blacklist in which an agent that is not applicable to the target website is recorded or a gray list in which an agent that is being used is recorded, from the available agent environment queue, if an available agent environment does not exist in the reusable agent environment queue.
Further, the second proxy environment selection element, further comprising:
a list screening unit capable of selecting an agent not in the blacklist or the grey list from the available agent queue;
and the environment construction unit can combine the proxy and the website information of the target website, and takes a combination result as a current proxy environment corresponding to the target website.
Further, the agent environment allocation module further includes an available queue retrieving component, where the available queue retrieving component includes:
the agent collecting unit can acquire a source agent from a third-party agent and place the source agent into an agent pool;
a rate detection unit capable of determining a response speed level of each of the source agents in the agent pool;
and an available agent queue building unit capable of storing the source agent into an available agent queue at regular time using the response speed level of each of the source agents as a storage priority.
A terminal according to an embodiment of the third aspect of the present invention is characterized in that the terminal includes: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the method for determining the reusable agent queue.
A computer-readable storage medium according to a fourth aspect of the present invention stores computer-executable instructions for performing a method for determining a reusable agent queue.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flowchart illustrating a method for determining a reusable agent queue according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for determining a reusable agent queue according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating the steps for determining available agent queues of FIG. 2;
FIG. 4 is a block diagram of an apparatus for determining a reusable agent queue according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar language means the same or similar program or programs having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
When the crawler program is used for acquiring data of the public website, the use of a single IP is limited due to the defense mechanism of the website, and the crawler program cannot acquire the data smoothly. Therefore, in order to enable the crawler program to run smoothly, the target website needs to be crawled through the proxy IP.
When a large number of agents are used, since the connection speed and status of each agent itself are different, it is necessary to uniformly manage the agents. However, in the existing scheme, the mode for managing a large number of proxy IPs is inefficient, and when a plurality of requests occur simultaneously, a thread security problem may occur.
In order to solve the problems in the prior art, a method for determining a reusable agent queue is provided.
The first embodiment,
Referring to fig. 1, the method includes at least the following steps:
and step S100, obtaining an access request from the client.
And step S200, determining the current agent environment suitable for the access request.
And selecting one proxy environment used as the current access request from the proxy environments.
Step S300, sending the access request to a target website based on the current proxy environment, and judging whether the current proxy environment is suitable for the target website according to the response message of the target website.
And judging whether the current agent environment is suitable for the target website or not according to the website response message obtained by accessing.
And S400, if the current agent environment is suitable for the target website, storing the current agent environment to a reusable agent environment queue corresponding to the target website.
If the current agent environment is suitable for the target website, the current agent environment is stored in the reusable agent environment queue, so that the steps of judgment can be reduced and the access efficiency can be accelerated in the subsequent process of accessing the target website.
Example II,
Referring to fig. 2, on the basis of the first embodiment, the method is described in more detail, and includes the following steps:
step S100, obtaining access request from client
And step S200, determining the current agent environment suitable for the access request. Specifically, the method comprises the following steps of,
step S201, determining a target website to be accessed by the access request, and acquiring a reusable agent environment queue corresponding to the target website.
And finding a target website needing to be accessed from the access request from the client, and finding a reusable agent environment queue corresponding to the target website. It is contemplated that the creation occurs if there is no reusable agent context queue corresponding to the target web site.
Step S202, if available agent environments exist in the reusable agent environment queue, selecting the agent environment from the reusable agent environment queue as the current agent environment.
When the reusable agent environment queue of the target website has the usable agent environment, the reusable agent environment queue which is determined to be capable of accessing the target website is directly selected to replace the agent environment, so that the access reliability is ensured, and the overall access efficiency is improved.
Step S203, if there is no available agent environment in the reusable agent environment queue, selecting an agent which is not in a blacklist or a grey list from the available agent queue as a current agent environment, wherein the blacklist records an agent which is not suitable for the target website, and the grey list records an agent which is being used.
When the agent environment cannot be obtained from the reusable agent environment queue, it indicates that the reusable agent environment is used up, and a backup mode needs to be provided to obtain the current agent environment, and the process of the mode includes:
a. selecting agents from the queue of available agents that are not on the blacklist or the grey list;
b. and combining the agent with the website information of the target website, and taking a combination result as a current agent environment corresponding to the target website.
The above steps are realized by selecting a free agent and then making the agent into an agent environment which can be used by the target website. Under the condition of high concurrency, certain accuracy can be sacrificed, and higher access efficiency is obtained.
Step S300, sending the access request to a target website based on the current proxy environment, and judging whether the current proxy environment is suitable for the target website according to the response message of the target website.
And after the access is finished, judging the connection state of the current agent environment and the target website according to a response message returned by the target website.
If the response status code in the response message is an environment failure status code, determining that the current agent environment is not applicable to the target website;
if the connection with the target website cannot be established through the current agent environment, judging that the current agent environment is not suitable for the target website;
if the connection with the target website can be established through the current agent environment, but the response of receiving the target website is overtime, judging that the current agent environment is not suitable for the target website;
if the response status code in the response message is the server abnormal status code, judging that the current agent environment is not suitable for the target website;
and if the other conditions exist, judging that the current agent environment is suitable for the target website.
According to some embodiments of the present application, when it is determined that the current proxy environment is not applicable to the present application, it is necessary to determine the reason for the inapplicability, because the reason for the inability of the proxy to access the server may be various, and may be either prohibited by the target website or a problem may occur in the proxy itself.
And detecting the connection state, and accessing the public network server through the current agent environment.
If the current agent environment can access the public network server, the agent environment is proved to have no problem and is shielded by the target website;
if the current agent environment can not establish network link with the public network server, the agent is dead, the agent is deleted from the available agent queue and written into the blacklist;
if the current agent environment is successfully connected with the public network server, but the response of the receiving server is overtime, the agent is judged to be dead, deleted from the available agent queue and written into a blacklist;
and if the response status code has an abnormal status code after the current agent environment establishes network connection with the public network server, judging that the agent is dead, deleting the agent from the available agent queue, and writing the agent into a blacklist.
It is understood that by public network server is meant an authoritative server that is accessible most of the time, such as a known search engine such as a hundred degrees (http:// www.baidu.com). And when visiting the public network, can also visit a plurality of public network servers simultaneously, reduce the possibility of erroneous judgement.
And S400, if the current agent environment is suitable for the target website, storing the current agent environment to a reusable agent environment queue corresponding to the target website.
And after the current agent environment is judged to be suitable for the target website, the current agent environment is stored in the reusable agent environment queue, so that the current agent environment can be directly used when being called again, the judgment process is reduced, and the access efficiency is improved.
According to some preferred embodiments of the present application, on the basis of the second embodiment, a step for determining available agent queues is provided.
Referring to fig. 3, there are no sequential steps as the determination of available agent queues is often done in conjunction with other steps. The method comprises the following steps:
step A100, obtaining a source agent from a third party agent and placing the source agent into an agent pool.
Step A200, determining the response speed level of each source agent in the agent pool.
Each source agent in the agent pool is connected with the public network server to ensure that the agent is available. And obtaining a response time for the origin agent to access the public network server.
If the response time is less than 1000 milliseconds, determining that the source agent is a high-quality agent;
if the response time is more than 1000 milliseconds and less than 3000 milliseconds, determining that the source agent is a middle-end agent;
and if the response time is more than 3000 milliseconds, determining that the source agent is a poor agent.
The number of seconds set here is a value set empirically, and generally speaking, a proxy capable of responding within three seconds is good in connection speed, and is often poor in quality more than three seconds.
Step A300, using the response speed level of each source agent as a storage priority, and storing the source agent into an available agent queue at regular time.
According to the response speed level in step a200, the high-quality agent and the middle-end agent are preferentially stored in the available agent queue, so that the proportion of the agents with poor quality can be reduced, and the processing efficiency of the whole method can be further increased.
Referring to fig. 4, a further embodiment of the present application provides an apparatus for determining a reusable agent queue, where the apparatus 20 includes an access request obtaining module 201, an agent environment allocating module 202, a response processing module 203, and a queue maintaining module 204.
An access request obtaining module 201 capable of obtaining an access request from a client;
a proxy environment assignment module 202 capable of determining a current proxy environment applicable to the access request;
the response processing module 203 is capable of sending the access request to a target website based on the current proxy environment, and determining whether the current proxy environment is suitable for the target website according to a response message of the target website;
and the queue maintenance module 204 is configured to, if the current proxy environment is applicable to the target website, store the current proxy environment to a reusable proxy environment queue corresponding to the target website.
The device 20 can determine whether the current proxy environment is suitable for the target website based on the website access result after the proxy environment is allocated to the client, and then store the proxy environment suitable for the target website into the reusable proxy queue by using the queue maintenance module, thereby increasing the access efficiency of the subsequent process.
Further, the agent environment allocation module further includes:
the queue selection element can determine a target website to be accessed by the access request and acquire a reusable agent environment queue corresponding to the target website;
a first agent environment selecting element, which selects an agent environment from the reusable agent environment queue as a current agent environment if an available agent environment exists in the reusable agent environment queue;
and a second agent environment selecting element, configured to select, as a current agent environment, an agent that is not on a blacklist in which an agent that is not applicable to the target website is recorded or a gray list in which an agent that is being used is recorded, from the available agent environment queue, if an available agent environment does not exist in the reusable agent environment queue.
Further, the second proxy environment selection element, further comprising:
a list screening unit capable of selecting an agent not in the blacklist or the grey list from the available agent queue;
and the environment construction unit can combine the proxy and the website information of the target website, and takes a combination result as a current proxy environment corresponding to the target website.
Further, the agent environment allocation module further includes an available queue retrieving component, where the available queue retrieving component includes:
the agent collecting unit can acquire a source agent from a third-party agent and place the source agent into an agent pool;
a rate detection unit capable of determining a response speed level of each of the source agents in the agent pool;
and an available agent queue building unit capable of storing the source agent into an available agent queue at regular time using the response speed level of each of the source agents as a storage priority.
Another embodiment of the present application provides a terminal, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the computer program implementing the method of determining a reusable agent queue as described above.
In particular, the processor may be a CPU, general purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. A processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, a DSP and a microprocessor, or the like.
In particular, the processor is coupled to the memory via a bus, which may include a path for communicating information. The bus may be a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc.
The memory may be, but is not limited to, a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, an EEPROM, a CD-ROM or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
Optionally, the memory is used for storing codes of computer programs for executing the scheme of the application, and the processor is used for controlling the execution. The processor is configured to execute application program code stored in the memory to implement any of the methods of fig. 1-4.
Yet another embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for performing the method for determining a reusable agent queue shown in fig. 1.
The above-described embodiments of the apparatus are merely illustrative, and the units illustrated as separate components may or may not be physically separate, may be located in one place, or may be distributed over a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
One of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
While the present invention has been described with reference to the preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method of determining a reusable agent queue, comprising the steps of:
acquiring an access request from a client;
determining a current agent environment applicable to the access request;
sending the access request to a target website based on the current proxy environment, and judging whether the current proxy environment is suitable for the target website according to a response message of the target website;
and if the current agent environment is suitable for the target website, storing the current agent environment to a reusable agent environment queue corresponding to the target website.
2. The method of claim 1, wherein the step of determining a current proxy context applicable to the access request comprises:
determining a target website to be accessed by the access request, and acquiring a reusable agent environment queue corresponding to the target website;
if available agent environments exist in the reusable agent environment queue, selecting the agent environment from the reusable agent environment queue as the current agent environment;
and if the reusable agent environment queue does not have available agent environments, selecting agents which are not in a blacklist or a grey list from the available agent queue as current agent environments, wherein the blacklist records agents which are not suitable for the target website, and the grey list records agents which are in use.
3. The method of claim 2, wherein the step of selecting from the queue of available agents an agent that is not on the blacklist or the grey list as the current agent context comprises:
selecting agents from the queue of available agents that are not on the blacklist or the grey list;
and combining the agent with the website information of the target website, and taking a combination result as a current agent environment corresponding to the target website.
4. The method of claim 2, wherein the step of determining the available agent queue comprises:
acquiring a source agent from a third party agent and placing the source agent into an agent pool;
determining a response speed level of each of the source agents in an agent pool;
and using the response speed level of each source agent as a storage priority to store the source agents into an available agent queue at regular time.
5. The method of claim 1, wherein the step of determining whether the current proxy environment is applicable to the target website according to the response message of the target website comprises:
if the response status code in the response message is an environment failure status code, determining that the current agent environment is not applicable to the target website;
if the connection with the target website cannot be established through the current agent environment, judging that the current agent environment is not suitable for the target website;
if the connection with the target website can be established through the current agent environment, but the response of receiving the target website is overtime, judging that the current agent environment is not suitable for the target website;
if the response status code in the response message is the server abnormal status code, judging that the current agent environment is not suitable for the target website;
and if the other conditions exist, judging that the current agent environment is suitable for the target website.
6. The method of claim 1, wherein after determining whether the current proxy environment is applicable to the target website according to the response message of the target website, further comprising:
and if the current agent environment is not suitable for the target website, detecting the self state of the current agent environment and correspondingly processing the current agent environment according to the detection result.
7. The method according to claim 5, wherein the step of detecting the self-state of the current agent environment and correspondingly processing the current agent environment according to the detection result if the current agent environment is not applicable to the target website specifically comprises:
accessing a public network server through the current proxy environment;
if the current agent environment can access the public network server, the current agent environment is not processed;
if the current agent environment can not establish network connection with a public network server, deleting the current agent environment from an available agent queue and writing the current agent environment into a blacklist;
if the current agent environment successfully establishes network connection with a public network server, but the response of a receiving server is overtime, deleting the current agent environment from an available agent queue and writing the current agent environment into a blacklist;
and if the response status code has an abnormal status code after the network connection between the current agent environment and the public network server is established, deleting the current agent environment from the available agent queue and writing the current agent environment into a blacklist.
8. An apparatus for determining a reusable agent queue, comprising:
the access request acquisition module can acquire an access request from a client;
the agent environment distribution module can determine the current agent environment applicable to the access request;
the response processing module can send the access request to a target website based on the current proxy environment and judge whether the current proxy environment is suitable for the target website according to the response message of the target website;
and the queue maintenance module is used for storing the current agent environment to a reusable agent environment queue corresponding to the target website if the current agent environment is suitable for the target website.
9. A terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method of any of claims 1 to 7.
10. A computer-readable storage medium storing computer-executable instructions for performing the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111142651.0A CN113905092B (en) | 2021-09-28 | 2021-09-28 | Method, device, terminal and storage medium for determining reusable agent queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111142651.0A CN113905092B (en) | 2021-09-28 | 2021-09-28 | Method, device, terminal and storage medium for determining reusable agent queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113905092A true CN113905092A (en) | 2022-01-07 |
| CN113905092B CN113905092B (en) | 2024-03-22 |
Family
ID=79029634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111142651.0A Active CN113905092B (en) | 2021-09-28 | 2021-09-28 | Method, device, terminal and storage medium for determining reusable agent queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113905092B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113923260A (en) * | 2021-09-28 | 2022-01-11 | 盐城金堤科技有限公司 | Method, device, terminal and storage medium for processing proxy environment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100172487A1 (en) * | 2009-01-08 | 2010-07-08 | Soundbite Communications, Inc. | Method and system for managing interactive communications campaign using a hold queue |
| CN103294732A (en) * | 2012-03-05 | 2013-09-11 | 富士通株式会社 | Web page crawling method and spider |
| US20160248836A1 (en) * | 2015-02-20 | 2016-08-25 | International Business Machines Corporation | Scalable self-healing architecture for client-server operations in transient connectivity conditions |
| US20170302526A1 (en) * | 2016-04-18 | 2017-10-19 | International Business Machines Corporation | Communication via a connection management message that uses an attribute having information on queue pair objects of a proxy node in a switchless network |
| CN107957999A (en) * | 2016-10-14 | 2018-04-24 | 北京国双科技有限公司 | A kind of web crawlers obtains the method and device of website data |
| CN108345642A (en) * | 2018-01-12 | 2018-07-31 | 深圳壹账通智能科技有限公司 | Method, storage medium and the server of website data are crawled using Agent IP |
| CN110851753A (en) * | 2019-11-07 | 2020-02-28 | 亿企赢网络科技有限公司 | Website access method, device, equipment and storage medium |
| CN110875899A (en) * | 2018-08-30 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data processing method, system and network system |
-
2021
- 2021-09-28 CN CN202111142651.0A patent/CN113905092B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100172487A1 (en) * | 2009-01-08 | 2010-07-08 | Soundbite Communications, Inc. | Method and system for managing interactive communications campaign using a hold queue |
| CN103294732A (en) * | 2012-03-05 | 2013-09-11 | 富士通株式会社 | Web page crawling method and spider |
| US20160248836A1 (en) * | 2015-02-20 | 2016-08-25 | International Business Machines Corporation | Scalable self-healing architecture for client-server operations in transient connectivity conditions |
| US20170302526A1 (en) * | 2016-04-18 | 2017-10-19 | International Business Machines Corporation | Communication via a connection management message that uses an attribute having information on queue pair objects of a proxy node in a switchless network |
| CN107957999A (en) * | 2016-10-14 | 2018-04-24 | 北京国双科技有限公司 | A kind of web crawlers obtains the method and device of website data |
| CN108345642A (en) * | 2018-01-12 | 2018-07-31 | 深圳壹账通智能科技有限公司 | Method, storage medium and the server of website data are crawled using Agent IP |
| CN110875899A (en) * | 2018-08-30 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Data processing method, system and network system |
| CN110851753A (en) * | 2019-11-07 | 2020-02-28 | 亿企赢网络科技有限公司 | Website access method, device, equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113923260A (en) * | 2021-09-28 | 2022-01-11 | 盐城金堤科技有限公司 | Method, device, terminal and storage medium for processing proxy environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113905092B (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11461242B2 (en) | Integration of application indicated minimum time to cache and maximum time to cache to least recently used track demoting schemes in a cache management system of a storage controller | |
| US8060920B2 (en) | Generating and changing credentials of a service account | |
| US10241681B2 (en) | Management of physical extents for space efficient storage volumes | |
| CN112650576B (en) | Resource scheduling method, device, equipment, storage medium and computer program product | |
| US11561905B2 (en) | Integration of application indicated minimum time to cache to least recently used track demoting schemes in a cache management system of a storage controller | |
| CN109981702B (en) | File storage method and system | |
| CN109213604B (en) | Data source management method and device | |
| WO2019170011A1 (en) | Task allocation method and device, and distributed storage system | |
| US10209905B2 (en) | Reusing storage blocks of a file system | |
| CN113905092B (en) | Method, device, terminal and storage medium for determining reusable agent queue | |
| CN110049084B (en) | Current limiting method, device and equipment of distributed system | |
| CN112631994A (en) | Data migration method and system | |
| US9954971B1 (en) | Cache eviction in a distributed computing system | |
| US20140245082A1 (en) | Implementing client based throttled error logging | |
| US20210382872A1 (en) | Blockchain-based Data Storage Method, Related Device and Storage Medium | |
| CN110837428A (en) | Storage device management method and device | |
| US20200028897A1 (en) | Load balancing system | |
| CN114363048A (en) | Mimicry unknown threat discovery system | |
| CN113923260B (en) | Method, device, terminal and storage medium for processing agent environment | |
| CN110113187B (en) | Configuration updating method and device, configuration server and configuration system | |
| US10254962B2 (en) | Asynchronously clearing page frames | |
| CN113300966A (en) | Flow control method, device and system and electronic equipment | |
| CN110955579A (en) | Ambari-based large data platform monitoring method | |
| KR102460910B1 (en) | Data storage method for preventing data duplication and data platform applying the same | |
| CN112637340B (en) | Domain name resolution system, monitoring method, cache cleaning method, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230801 Address after: Room 404-405, 504, Building B-17-1, Big data Industrial Park, Kecheng Street, Yannan High tech Zone, Yancheng, Jiangsu Province, 224000 Applicant after: Yancheng Tianyanchawei Technology Co.,Ltd. Address before: 224000 room 501-503, building b-17-1, Xuehai road big data Industrial Park, Kecheng street, Yannan high tech Zone, Yancheng City, Jiangsu Province Applicant before: Yancheng Jindi Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |