CN111355693B - Proxy service realization method, device, electronic equipment and storage medium - Google Patents
Proxy service realization method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111355693B CN111355693B CN201811581126.7A CN201811581126A CN111355693B CN 111355693 B CN111355693 B CN 111355693B CN 201811581126 A CN201811581126 A CN 201811581126A CN 111355693 B CN111355693 B CN 111355693B
- Authority
- CN
- China
- Prior art keywords
- proxy
- target equipment
- equipment
- service
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000012795 verification Methods 0.000 claims abstract description 26
- 230000005540 biological transmission Effects 0.000 claims abstract description 18
- 230000009193 crawling Effects 0.000 claims description 14
- 230000008569 process Effects 0.000 abstract description 23
- 230000008859 change Effects 0.000 abstract description 7
- 230000006872 improvement Effects 0.000 abstract description 5
- 238000012216 screening Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0236—Filtering by address, protocol, port number or service, e.g. IP-address or URL
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0281—Proxies
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/083—Network architectures or network communication protocols for network security for authentication of entities using passwords
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0876—Network architectures or network communication protocols for network security for authentication of entities based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint
Landscapes
- Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Power Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a proxy service realization method, a proxy service realization device, electronic equipment and a storage medium. The method comprises the following steps: receiving a service request containing verification information; determining target equipment serving as a proxy outlet according to the verification information, and establishing proxy connection with the target equipment; and realizing the transmission of service data based on the proxy connection. On the one hand, authentication information is utilized to perform normal authentication of proxy service, on the other hand, target equipment serving as a proxy outlet can be determined according to the authentication information, so that when service data transmission is realized based on proxy connection established with the target equipment, the fixed target equipment can be used as the proxy outlet, and the unstable access process caused by the change of the outlet ip of the access process is avoided; the target equipment can be determined by using the verification information, advanced screening can be realized, excessive improvement on the original proxy protocol is not needed, a new functional module is not needed to be built, the conception is ingenious, the application is wide, the existing network architecture is not needed to be changed greatly, and the method is quite convenient.
Description
Technical Field
The present invention relates to the field of network technologies, and in particular, to a method and apparatus for implementing proxy service, an electronic device, and a storage medium.
Background
A proxy service is a special network service that allows one network terminal to make an indirect connection with another network terminal through this service. Some network devices such as gateways and routers have network proxy functions. Proxy services are generally considered to be beneficial for guaranteeing the privacy or security of network terminals and preventing attacks.
At present, when a network terminal is connected with another network terminal through a proxy server, larger randomness exists, so that a plurality of network outlets are adopted in one network access process, and inconvenience is caused. For example, in accessing a designated web site using a proxy service, access is performed using a plurality of IP addresses, and the designated web site is easily determined to be an illegal access behavior.
Disclosure of Invention
The present invention has been made in view of the above problems, and provides a method, apparatus, electronic device, and storage medium for implementing a proxy service that overcomes or at least partially solves the above problems.
According to one aspect of the present invention, there is provided a method for implementing a proxy service, including:
receiving a service request containing verification information;
determining target equipment serving as a proxy outlet according to the verification information, and establishing proxy connection with the target equipment;
and realizing the transmission of service data based on the proxy connection.
Optionally, the service request is a service request related to a crawler;
the transmission of the service data based on the proxy connection comprises the following steps:
and taking the IP address of the target equipment as the IP of the outlet of the crawler, and transmitting the crawling request and the crawled content through the proxy connection.
Optionally, the authentication information includes a user name and a password, and the determining the target device serving as the proxy outlet according to the authentication information includes:
analyzing the user name, and judging whether new target equipment needs to be allocated according to an analysis result;
and if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
Optionally, the parsing the user name includes:
resolving a session name from the user name;
the judging whether new target equipment needs to be allocated according to the analysis result comprises the following steps:
judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
Optionally, the establishing a proxy connection with the target device includes:
and establishing a proxy session according to the session name, and recording the session name and the corresponding target equipment.
Optionally, the proxy mode includes: a random mode and a binding mode;
the device for selecting matching according to the proxy mode corresponding to the analysis result as the target device comprises:
in the random mode, the random distribution device serves as a target device;
in the binding mode, the designated device is taken as the target device.
Optionally, the parsing the user name includes: analyzing the service requirement or the equipment information of the appointed equipment from the user name;
the random distribution device as a target device includes: and randomly allocating the equipment meeting the service requirement as target equipment.
Optionally, the proxy service is implemented based on the socks5 protocol.
According to another aspect of the present invention, there is provided an implementation apparatus of a proxy service, including:
a receiving unit adapted to receive a service request containing authentication information;
a proxy connection establishment unit adapted to determine a target device serving as a proxy outlet according to the authentication information, and establish a proxy connection with the target device;
and the proxy unit is suitable for realizing the transmission of service data based on the proxy connection.
Optionally, the service request is a service request related to a crawler;
the proxy unit is suitable for transmitting crawling requests and crawled contents through the proxy connection by taking the IP address of the target device as the outlet IP of the crawler.
Optionally, the authentication information includes a user name and a password;
the proxy connection establishment unit is suitable for analyzing the user name and judging whether new target equipment needs to be allocated according to an analysis result; and if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
Optionally, the proxy connection establishment unit is adapted to parse a session name from the user name; judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
Optionally, the proxy connection establishment unit is adapted to establish a proxy session according to the session name, and record the session name and the corresponding target device.
Optionally, the proxy mode includes: a random mode and a binding mode;
the proxy connection establishment unit is suitable for randomly distributing equipment as target equipment in a random mode; in the binding mode, the designated device is taken as the target device.
Optionally, the proxy connection establishment unit is adapted to parse service requirements or device information of a designated device from the user name; and randomly allocating the equipment meeting the service requirement as target equipment.
Optionally, the proxy service is implemented based on the socks5 protocol.
According to still another aspect of the present invention, there is provided an electronic apparatus including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform a method as described in any of the above.
According to a further aspect of the present invention there is provided a computer readable storage medium storing one or more programs which when executed by a processor implement a method as described in any of the above.
As can be seen from the above, after receiving a service request containing verification information, the technical solution of the present invention performs authentication of normal proxy service by using the verification information, and on the other hand, the target device used as a proxy outlet can be determined according to the verification information, so that when transmission of service data is implemented based on a proxy connection established with the target device, the fixed target device can be used as the proxy outlet, thereby avoiding the occurrence of a change in the outlet ip of the access process and unstable access process; the target equipment can be determined by using the verification information, and advanced screening can be realized, for example, target equipment of a certain province, a certain city, a certain type and the like are selected, excessive improvement on an original proxy protocol is not needed, a new functional module is not needed to be built, the conception is ingenious, the application is wide, the existing network architecture is not needed to be changed greatly, and the method is quite convenient.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 illustrates a flow diagram of a method of implementing a proxy service according to one embodiment of the invention;
FIG. 2 is a schematic diagram of a proxy service implementation device according to one embodiment of the present invention;
FIG. 3 shows a schematic diagram of an electronic device according to one embodiment of the invention;
fig. 4 illustrates a schematic structure of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 shows a flow diagram of a method for implementing a proxy service according to an embodiment of the invention. As shown in fig. 1, the method includes:
step S110, a service request containing authentication information is received.
Here, the authentication information may include a user name and a password for authentication in the proxy service process.
Step S120, determining a target device serving as a proxy outlet according to the verification information, and establishing proxy connection with the target device.
In this step, the determination of the target device is achieved using the authentication information. That is, in an embodiment of the present invention, the authentication information may be in a specific format, not only for authentication, but also for determination of the target device. The subsequent establishment of the proxy connection may be implemented with reference to the prior art, and will not be described in detail herein.
Step S130, the transmission of service data is realized based on the proxy connection.
It can be seen that, in the method shown in fig. 1, after receiving a service request containing verification information, on the one hand, authentication information is utilized to perform authentication of normal proxy service, and on the other hand, a target device used as a proxy outlet can be determined according to the verification information, so that when transmission of service data is realized based on proxy connection established with the target device, the fixed target device can be used as the proxy outlet, and the unstable access process caused by the change of the outlet ip of the access process is avoided; the target equipment can be determined by using the verification information, and advanced screening can be realized, for example, target equipment of a certain province, a certain city, a certain type and the like are selected, excessive improvement on an original proxy protocol is not needed, a new functional module is not needed to be built, the conception is ingenious, the application is wide, the existing network architecture is not needed to be changed greatly, and the method is quite convenient.
In one embodiment of the present invention, in the above method, the service request is a service request related to a crawler; the transmission of service data based on proxy connection comprises: and taking the IP address of the target equipment as the IP of the outlet of the crawler, and transmitting the crawling request and the crawled content through proxy connection.
Crawlers are an important tool for acquiring network resources and are one of the core technologies of search engines. Crawlers often require multiple network outlets because access behavior is significantly different from normal manual access, where it is often necessary to connect different devices as network outlets by means of proxy services. In this process, there may be a problem pointed out in the background of the invention due to the number of optional devices: in a crawling process, multiple requests may be generated, such as crawling resources of several different pages of a website; however, for this multiple request, different devices are likely to be randomly allocated, resulting in a change in the exit IP, which is likely to be abnormal for the crawled web site, resulting in a crawl failure.
Therefore, in a specific embodiment of the present invention, a fixed target device is used as an agent exit and an IP address of the target device is used as an exit IP of a crawler in a session process, so that the occurrence of the above situation can be effectively avoided, and the stability and reliability of the service are ensured.
In one embodiment of the present invention, in the above method, the authentication information includes a user name and a password, and determining the target device serving as the proxy outlet according to the authentication information includes: analyzing the user name, and judging whether new target equipment needs to be allocated according to the analysis result; if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
In a specific embodiment of the present invention, the user name may be divided into a plurality of parts, where a part of content includes information related to allocation of target devices, and when the user name is parsed, it may be determined whether new target devices need to be allocated according to the parsing result if the part of content is parsed, if not, it is indicated that the bound target devices can be used, and if so, a matching device is selected according to the corresponding proxy mode.
In an embodiment of the present invention, in the above method, resolving the user name includes: analyzing the session name from the user name; judging whether new target equipment needs to be allocated according to the analysis result comprises the following steps: and judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
In one embodiment of the present invention, the session name may be Rand-12345, with the front Rand corresponding to the random pattern in the proxy mode and the rear 12345 being the session name. If the session name already has the corresponding target equipment, the target equipment is still used; otherwise, the target device needs to be found. In one embodiment of the present invention, in the method, establishing a proxy connection with the target device includes: and establishing the proxy session according to the session name, and recording the session name and the corresponding target equipment.
In one embodiment of the present invention, in the method, the proxy mode includes: a random mode and a binding mode; the selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result comprises the following steps: in the random mode, the random distribution device serves as a target device; in the binding mode, the designated device is taken as the target device.
In a specific embodiment, the random allocation may randomly choose one target device from all available target devices as a proxy exit, and the binding mode designates a certain device as a proxy exit.
Of course, the random allocation may also follow a certain rule, and in one embodiment of the present invention, the parsing the user name includes: analyzing the service requirement or the equipment information of the appointed equipment from the user name; the random distribution device includes, as a target device: and randomly allocating the equipment meeting the service requirements as target equipment.
For example, a province, a city, or a type of target device may be selected, and only the province, city, or device type information may be written to the user name. For the binding mode, device information specifying the device needs to be written in the user name.
In the embodiment of the invention, the appointed equipment can be an internet of things (IoT) equipment, namely intelligent equipment in the internet of things, so that the internet of things resource is effectively utilized; the server side only needs to define the format of the user name and can accurately analyze the necessary information from the user name. The proxy protocol used may be a protocol common in the prior art, requiring no new functional modules or excessive modifications.
For example, in one embodiment of the present invention, in the above method, the proxy service is implemented based on the socks5 protocol. The socks5 can play an intermediary role between the front-end machine and the server machine which communicate by using the TCP/IP protocol, so that the front-end machine in the intranet can access the server in the Internet network, or the communication is safer.
In a specific embodiment, the crawler program set on the server side is connected with the proxy service in the transfer server, and finally establishes proxy connection with the target device to perform crawling, so that a business process is realized.
Fig. 2 is a schematic structural diagram of a proxy service implementation apparatus according to an embodiment of the present invention. As shown in fig. 2, the proxy service implementation apparatus 200 includes:
the receiving unit 210 is adapted to receive a service request comprising authentication information.
Here, the authentication information may include a user name and a password for authentication in the proxy service process.
The proxy connection establishment unit 220 is adapted to determine a target device serving as a proxy outlet based on the authentication information, and to establish a proxy connection with the target device.
The proxy connection setup unit 220 uses the authentication information to achieve determination of the target device. That is, in an embodiment of the present invention, the authentication information may be in a specific format, not only for authentication, but also for determination of the target device. The subsequent establishment of the proxy connection may be implemented with reference to the prior art, and will not be described in detail herein.
The proxy unit 230 is adapted to implement transmission of service data based on the proxy connection.
It can be seen that, after receiving the service request including the verification information, the apparatus shown in fig. 2 performs authentication of the normal proxy service by using the verification information, and on the other hand, the apparatus can determine the target device used as the proxy exit according to the verification information, so that when implementing transmission of service data based on the proxy connection established with the apparatus, the apparatus can be used as the proxy exit through the fixed target device, thereby avoiding the change of the exit ip in the access process and the instability of the access process; the target equipment can be determined by using the verification information, and advanced screening can be realized, for example, target equipment of a certain province, a certain city, a certain type and the like are selected, excessive improvement on an original proxy protocol is not needed, a new functional module is not needed to be built, the conception is ingenious, the application is wide, the existing network architecture is not needed to be changed greatly, and the method is quite convenient.
In one embodiment of the present invention, in the foregoing apparatus, the service request is a service request related to a crawler; the proxy unit 230 is adapted to transmit the crawling request and the crawled content over a proxy connection with the IP address of the target device as the outlet IP of the crawler.
Crawlers are an important tool for acquiring network resources and are one of the core technologies of search engines. Crawlers often require multiple network outlets because access behavior is significantly different from normal manual access, where it is often necessary to connect different devices as network outlets by means of proxy services. In this process, there may be a problem pointed out in the background of the invention due to the number of optional devices: in a crawling process, multiple requests may be generated, such as crawling resources of several different pages of a website; however, for this multiple request, different devices are likely to be randomly allocated, resulting in a change in the exit IP, which is likely to be abnormal for the crawled web site, resulting in a crawl failure.
Therefore, in a specific embodiment of the present invention, a fixed target device is used as an agent exit and an IP address of the target device is used as an exit IP of a crawler in a session process, so that the occurrence of the above situation can be effectively avoided, and the stability and reliability of the service are ensured.
In one embodiment of the present invention, in the above apparatus, the authentication information includes a user name and a password; the proxy connection establishment unit 220 is adapted to parse the user name, and determine whether a new target device needs to be allocated according to the parsing result; if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
In a specific embodiment of the present invention, the user name may be divided into a plurality of parts, where a part of content includes information related to allocation of target devices, and when the user name is parsed, it may be determined whether new target devices need to be allocated according to the parsing result if the part of content is parsed, if not, it is indicated that the bound target devices can be used, and if so, a matching device is selected according to the corresponding proxy mode.
In one embodiment of the present invention, in the above apparatus, the proxy connection establishment unit 220 is adapted to parse a session name from a user name; and judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
In one embodiment of the present invention, the session name may be Rand-12345, with the front Rand corresponding to the random pattern in the proxy mode and the rear 12345 being the session name. If the session name already has the corresponding target equipment, the target equipment is still used; otherwise, the target device needs to be found. In an embodiment of the present invention, in the above apparatus, the proxy connection establishing unit 220 is adapted to establish a proxy session according to a session name, and record the session name and a corresponding target device.
In one embodiment of the present invention, in the foregoing apparatus, the proxy mode includes: a random mode and a binding mode; a proxy connection establishment unit 220 adapted to randomly allocate a device as a target device in a random mode; in the binding mode, the designated device is taken as the target device.
In a specific embodiment, the random allocation may randomly choose one target device from all available target devices as a proxy exit, and the binding mode designates a certain device as a proxy exit.
Of course, the random allocation may also follow a certain rule, and in one embodiment of the present invention, in the above apparatus, the proxy connection establishment unit 220 is adapted to parse the service requirement or the device information of the designated device from the user name; and randomly allocating the equipment meeting the service requirements as target equipment.
For example, a province, a city, or a type of target device may be selected, and only the province, city, or device type information may be written to the user name. For the binding mode, device information specifying the device needs to be written in the user name.
In the embodiment of the invention, the appointed equipment can be an internet of things (IoT) equipment, namely intelligent equipment in the internet of things, so that the internet of things resource is effectively utilized; the server side only needs to define the format of the user name and can accurately analyze the necessary information from the user name. The proxy protocol used may be a protocol common in the prior art, requiring no new functional modules or excessive modifications.
For example, in one embodiment of the present invention, in the above apparatus, the proxy service is implemented based on the socks5 protocol. The socks5 can play an intermediary role between the front-end machine and the server machine which communicate by using the TCP/IP protocol, so that the front-end machine in the intranet can access the server in the Internet network, or the communication is safer.
In a specific embodiment, the crawler program set on the server side is connected with the proxy service in the transfer server, and finally establishes proxy connection with the target device to perform crawling, so that a business process is realized.
In summary, after receiving a service request including verification information, the technical solution of the present invention performs authentication of normal proxy service by using the verification information, and on the other hand, the target device used as a proxy exit can be determined according to the verification information, so that when transmission of service data is implemented based on proxy connection established with the target device, the target device can be used as a proxy exit, and the occurrence of a change in the exit ip of an access process, which leads to instability of the access process, is avoided; the target equipment can be determined by using the verification information, and advanced screening can be realized, for example, target equipment of a certain province, a certain city, a certain type and the like are selected, excessive improvement on an original proxy protocol is not needed, a new functional module is not needed to be built, the conception is ingenious, the application is wide, the existing network architecture is not needed to be changed greatly, and the method is quite convenient.
It should be noted that:
the algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose devices may also be used with the teachings herein. The required structure for the construction of such devices is apparent from the description above. In addition, the present invention is not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.
Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in a proxy service implementing apparatus according to an embodiment of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.
For example, fig. 3 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device comprises a processor 310 and a memory 320 arranged to store computer executable instructions (computer readable program code). The memory 320 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. The memory 320 has a memory space 330 storing computer readable program code 331 for performing any of the method steps described above. For example, the memory space 330 for storing computer readable program code may include respective computer readable program code 331 for implementing the respective steps in the above method, respectively. The computer readable program code 331 can be read from or written to one or more computer program products. These computer program products comprise a program code carrier such as a hard disk, a Compact Disc (CD), a memory card or a floppy disk. Such a computer program product is typically a computer readable storage medium as described for example in fig. 4. Fig. 4 illustrates a schematic structure of a computer-readable storage medium according to an embodiment of the present invention. The computer readable storage medium 400 stores computer readable program code 331 for performing the steps of the method according to the invention, which may be read by the processor 310 of the electronic device 300, which computer readable program code 331, when executed by the electronic device 300, causes the electronic device 300 to perform the steps of the method described above, in particular the computer readable program code 331 stored by the computer readable storage medium may perform the method shown in any of the embodiments described above. The computer readable program code 331 may be compressed in a suitable form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.
The embodiment of the invention discloses A1, a realization method of proxy service, comprising the following steps:
receiving a service request containing verification information;
determining target equipment serving as a proxy outlet according to the verification information, and establishing proxy connection with the target equipment;
and realizing the transmission of service data based on the proxy connection.
A2, the method of A1, wherein the service request is a service request related to a crawler;
the transmission of the service data based on the proxy connection comprises the following steps:
and taking the IP address of the target equipment as the IP of the outlet of the crawler, and transmitting the crawling request and the crawled content through the proxy connection.
A3, the method of A1, wherein the authentication information includes a user name and a password, and the determining a target device serving as a proxy outlet according to the authentication information includes:
analyzing the user name, and judging whether new target equipment needs to be allocated according to an analysis result;
and if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
A4, the method of A3, wherein the analyzing the user name comprises:
resolving a session name from the user name;
the judging whether new target equipment needs to be allocated according to the analysis result comprises the following steps:
judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
The method of A5, A4, wherein the establishing a proxy connection with the target device comprises:
and establishing a proxy session according to the session name, and recording the session name and the corresponding target equipment.
A6. the method of A3, wherein the proxy mode includes: a random mode and a binding mode;
the device for selecting matching according to the proxy mode corresponding to the analysis result as the target device comprises:
in the random mode, the random distribution device serves as a target device;
in the binding mode, the designated device is taken as the target device.
A7, the method of A6, wherein the analyzing the user name comprises: analyzing the service requirement or the equipment information of the appointed equipment from the user name;
the random distribution device as a target device includes: and randomly allocating the equipment meeting the service requirement as target equipment.
A8, the method of A1, wherein the proxy service is implemented based on a socks5 protocol.
The embodiment of the invention also discloses a B9 and a device for realizing the proxy service, which comprises the following steps:
a receiving unit adapted to receive a service request containing authentication information;
a proxy connection establishment unit adapted to determine a target device serving as a proxy outlet according to the authentication information, and establish a proxy connection with the target device;
and the proxy unit is suitable for realizing the transmission of service data based on the proxy connection.
B10, the device of B9, wherein the service request is a service request related to a crawler;
the proxy unit is suitable for transmitting crawling requests and crawled contents through the proxy connection by taking the IP address of the target device as the outlet IP of the crawler.
B11, the apparatus of B9, wherein the authentication information includes a user name and a password;
the proxy connection establishment unit is suitable for analyzing the user name and judging whether new target equipment needs to be allocated according to an analysis result; and if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
B12, the device of B11, wherein,
the proxy connection establishment unit is suitable for resolving a session name from the user name; judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
B13, the device of B12, wherein,
the proxy connection establishing unit is suitable for establishing a proxy session according to the session name and recording the session name and the corresponding target equipment.
B14, the apparatus of B11, wherein the proxy mode includes: a random mode and a binding mode;
the proxy connection establishment unit is suitable for randomly distributing equipment as target equipment in a random mode; in the binding mode, the designated device is taken as the target device.
The device of B15, B14, wherein,
the proxy connection establishment unit is suitable for resolving service requirements or equipment information of appointed equipment from the user name; and randomly allocating the equipment meeting the service requirement as target equipment.
B16, the apparatus of B9, wherein the proxy service is implemented based on a socks5 protocol.
The embodiment of the invention also discloses C17 and electronic equipment, wherein the electronic equipment comprises: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the method of any of A1-A8.
Embodiments of the invention also disclose D18, a computer readable storage medium storing one or more programs which, when executed by a processor, implement the method of any of A1-A8.
Claims (18)
1. A method for implementing proxy services, comprising:
receiving a service request containing verification information;
determining target equipment serving as a proxy outlet according to the verification information, and establishing proxy connection with the target equipment;
realizing transmission of service data based on the proxy connection;
the authentication information includes a user name and a password, and the determining a target device serving as a proxy outlet according to the authentication information includes: and analyzing the user name, and judging whether new target equipment needs to be allocated according to an analysis result.
2. The method of claim 1, wherein the business request is a business request related to a crawler;
the transmission of the service data based on the proxy connection comprises the following steps:
and taking the IP address of the target equipment as the IP of the outlet of the crawler, and transmitting the crawling request and the crawled content through the proxy connection.
3. The method of claim 1, wherein after determining whether a new target device needs to be allocated according to the parsing result, further comprising:
and if new target equipment needs to be distributed, selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result.
4. The method of claim 1, wherein the resolving the user name comprises:
resolving a session name from the user name;
the judging whether new target equipment needs to be allocated according to the analysis result comprises the following steps:
judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
5. The method of claim 4, wherein the establishing a proxy connection with the target device comprises:
and establishing a proxy session according to the session name, and recording the session name and the corresponding target equipment.
6. The method of claim 3, wherein the proxy mode comprises: a random mode and a binding mode;
the device for selecting matching according to the proxy mode corresponding to the analysis result as the target device comprises:
in the random mode, the random distribution device serves as a target device;
in the binding mode, the designated device is taken as the target device.
7. The method of claim 6, wherein the resolving the user name comprises: analyzing the service requirement or the equipment information of the appointed equipment from the user name;
the random distribution device as a target device includes: and randomly allocating the equipment meeting the service requirement as target equipment.
8. The method of claim 1, wherein the proxy service is implemented based on a socks5 protocol.
9. An implementation apparatus of proxy service, comprising:
a receiving unit adapted to receive a service request containing authentication information;
a proxy connection establishment unit adapted to determine a target device serving as a proxy outlet according to the authentication information, and establish a proxy connection with the target device;
the proxy unit is suitable for realizing transmission of service data based on the proxy connection;
when the verification information comprises a user name and a password, the proxy connection establishment unit is suitable for analyzing the user name and judging whether new target equipment needs to be allocated according to an analysis result.
10. The apparatus of claim 9, wherein the business request is a business request related to a crawler;
the proxy unit is suitable for transmitting crawling requests and crawled contents through the proxy connection by taking the IP address of the target device as the outlet IP of the crawler.
11. The apparatus of claim 9, wherein,
the proxy connection establishing unit is suitable for selecting matched equipment as target equipment according to the proxy mode corresponding to the analysis result if new target equipment needs to be distributed.
12. The apparatus of claim 11, wherein,
the proxy connection establishment unit is suitable for resolving a session name from the user name; judging whether the target equipment corresponding to the session name is recorded, if so, not needing to be allocated with new target equipment, otherwise, needing to be allocated with new target equipment.
13. The apparatus of claim 12, wherein,
the proxy connection establishing unit is suitable for establishing a proxy session according to the session name and recording the session name and the corresponding target equipment.
14. The apparatus of claim 11, wherein the proxy mode comprises: a random mode and a binding mode;
the proxy connection establishment unit is suitable for randomly distributing equipment as target equipment in a random mode; in the binding mode, the designated device is taken as the target device.
15. The apparatus of claim 14, wherein,
the proxy connection establishment unit is suitable for resolving service requirements or equipment information of appointed equipment from the user name; and randomly allocating the equipment meeting the service requirement as target equipment.
16. The apparatus of claim 9, wherein the proxy service is implemented based on a socks5 protocol.
17. An electronic device, wherein the electronic device comprises: a processor; and a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method of any of claims 1-8.
18. A computer readable storage medium, wherein the computer readable storage medium stores one or more programs, which when executed by a processor, implement the method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811581126.7A CN111355693B (en) | 2018-12-24 | 2018-12-24 | Proxy service realization method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811581126.7A CN111355693B (en) | 2018-12-24 | 2018-12-24 | Proxy service realization method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111355693A CN111355693A (en) | 2020-06-30 |
| CN111355693B true CN111355693B (en) | 2023-10-31 |
Family
ID=71193749
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811581126.7A Active CN111355693B (en) | 2018-12-24 | 2018-12-24 | Proxy service realization method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111355693B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891858A (en) * | 2012-10-22 | 2013-01-23 | 北京瓦力网络科技有限公司 | Method and system for network data transmission and proxy server |
| CN107770138A (en) * | 2016-08-22 | 2018-03-06 | 阿里巴巴集团控股有限公司 | Specify the method and proxy server, client of IP address |
| CN108063714A (en) * | 2016-11-09 | 2018-05-22 | 北京国双科技有限公司 | A kind of processing method and processing device of network request |
| CN108551452A (en) * | 2018-04-18 | 2018-09-18 | 平安科技（深圳）有限公司 | Web crawlers method, terminal and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7953868B2 (en) * | 2007-01-31 | 2011-05-31 | International Business Machines Corporation | Method and system for preventing web crawling detection |
| US8799470B2 (en) * | 2011-03-11 | 2014-08-05 | Qualcomm Incorporated | System and method using a client-local proxy-server to access a device having an assigned network address |
| CN102708173B (en) * | 2012-05-02 | 2014-08-13 | 北京奇虎科技有限公司 | Method and system for processing user requests of accessing to web pages |
| CN103914568B (en) * | 2014-04-24 | 2017-05-03 | 厦门市美亚柏科信息股份有限公司 | Method and device for dispatching HTTP proxy |
| CN106612261A (en) * | 2015-10-26 | 2017-05-03 | 北京国双科技有限公司 | Website data obtaining method, devices and system |
| CN106021375B (en) * | 2016-05-11 | 2019-11-26 | 深圳市永兴元科技股份有限公司 | Data bank access method and database broker node |
-
2018
- 2018-12-24 CN CN201811581126.7A patent/CN111355693B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102891858A (en) * | 2012-10-22 | 2013-01-23 | 北京瓦力网络科技有限公司 | Method and system for network data transmission and proxy server |
| CN107770138A (en) * | 2016-08-22 | 2018-03-06 | 阿里巴巴集团控股有限公司 | Specify the method and proxy server, client of IP address |
| CN108063714A (en) * | 2016-11-09 | 2018-05-22 | 北京国双科技有限公司 | A kind of processing method and processing device of network request |
| CN108551452A (en) * | 2018-04-18 | 2018-09-18 | 平安科技（深圳）有限公司 | Web crawlers method, terminal and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111355693A (en) | 2020-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104158818B (en) | A kind of single-point logging method and system | |
| CN108259425A (en) | The determining method, apparatus and server of query-attack | |
| CN109088909B (en) | Service gray level publishing method and device based on merchant type | |
| CN108833450B (en) | Method and device for preventing server from being attacked | |
| CN104219230B (en) | Identify method and the device of malicious websites | |
| CN108241797A (en) | Mirror image warehouse user right management method, device, system and readable storage medium storing program for executing | |
| CN107239701B (en) | Method and device for identifying malicious website | |
| CN109657434B (en) | Application access method and device | |
| JP6666441B2 (en) | IP address obtaining method and apparatus | |
| CN108718337A (en) | Website account login, verification, verification information processing method, apparatus and system | |
| CN108132973A (en) | Loading method and device, the computer readable storage medium of WEB jump page | |
| CN113098835A (en) | Honeypot implementation method based on block chain, honeypot client and honeypot system | |
| CN111355693B (en) | Proxy service realization method, device, electronic equipment and storage medium | |
| CN111225038B (en) | Server access method and device | |
| CN115333863B (en) | Internet of things system building method based on dynamic domain name service and related equipment | |
| CN104009999A (en) | Method and device for preventing ARP cheating and network access server | |
| CN105959248B (en) | The method and device of message access control | |
| CN109451094B (en) | Method, system, electronic device and medium for acquiring IP address of source station | |
| CN113472545B (en) | Equipment network access method, device, equipment, storage medium and communication system | |
| CN112468356B (en) | Router interface testing method, device, electronic equipment and storage medium | |
| US11729246B2 (en) | Apparatus and method for determining types of uniform resource locator | |
| CN111885063B (en) | Open source system access control method, device, equipment and storage medium | |
| CN112637106B (en) | Method and device for terminal to access website | |
| CN114244555A (en) | Method for adjusting security policy | |
| CN106559420A (en) | A kind of filter method and device of message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |