CN111818038B

CN111818038B - Network data acquisition and identification method and device

Info

Publication number: CN111818038B
Application number: CN202010627703.2A
Authority: CN
Inventors: 范晓锋; 方庆远
Original assignee: Lazas Network Technology Shanghai Co Ltd
Current assignee: Lazas Network Technology Shanghai Co Ltd
Priority date: 2020-07-01
Filing date: 2020-07-01
Publication date: 2023-01-31
Anticipated expiration: 2040-07-01
Also published as: CN111818038A

Abstract

The method comprises the steps of decrypting a target encryption parameter carried by a second network request to obtain a first network behavior background mark, obtaining a second network behavior background mark corresponding to the second network request, comparing the first network behavior background mark with the second network behavior background mark, and determining whether an access main body corresponding to the second network request is a network data acquisition main body or not based on a comparison result, wherein the second network request is realized based on network response data corresponding to the first network request, and the target encryption parameter is a parameter obtained after encryption processing is carried out on the first network behavior background mark corresponding to the first network request. By using the method, the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy distribution mechanism can be accurately identified.

Description

Network data acquisition and identification method and device

Technical Field

The application relates to the technical field of computers, in particular to a network data acquisition and identification method. The application also relates to a network data acquisition and identification device, an electronic device and a computer readable storage medium.

Background

The network data acquisition refers to automatically acquiring world wide web information through a program or a script according to a preset data acquisition rule, and the process may cause network resources, computing resources and the like of a website to be improperly occupied, so that the function of the website is damaged. For example, in an online shopping scenario or a life service type network application scenario, the network data acquisition process may cause that merchant information, commodity information, and the like of a network platform are maliciously acquired in batches; for a content distribution website, it may cause the intellectual property rights of the website to be maliciously infringed.

The network data acquisition main body can usually acquire target data content through an HTTP request, corresponding parameters in a URL address (Uniform Resource Locator) of the HTTP request correspond to the target data content, and the network data acquisition main body continuously extracts a new URL from a URL corresponding to one or a plurality of initial webpages and puts the new URL into an acquisition queue according to a webpage acquisition policy after acquiring the initial webpage URL, and acquires the target data content based on the URL until a certain stop condition is satisfied.

The existing identification and interception technology aiming at the network data acquisition behavior comprises modes of IP-based dimensional confrontation, account dimensional confrontation, verification codes, dynamic page structures, access amount and download amount limitation, text-to-image conversion, dynamic parameters and the like, wherein the identification and interception technology aiming at the network data acquisition behavior based on the dynamic parameters increases the cost of the network data acquisition behavior to a certain extent and increases the opportunity of identifying and intercepting the network data acquisition behavior. For example, the takeaway website performs dynamic processing on the URL corresponding to the merchant, which is specifically to differentiate the merchant ID in the URL corresponding to the merchant into a merchant internal ID and an external dynamic ID, where the internal ID is a transfer parameter used for specifying a merchant object between systems inside the website and is generally a fixed character string, and the external dynamic ID is a dynamic character string obtained by mapping numbers and changing at regular or irregular time, so that the network data acquisition main body cannot request data content of the merchant from the website through the same static URL, but needs to find an exact value of the dynamic URL corresponding to the merchant at different time points, for example, by searching the name of the merchant in the website, the manner increases the number of times that the network data acquisition main body interacts with the website to acquire a target data content, increases the acquisition cost, and increases the opportunity of identifying and intercepting network data acquisition behaviors by the website.

In order to resist the identification and interception technology aiming at the network data acquisition behavior based on the dynamic parameters and realize the efficient traversal of the whole content of the website, a task scheduling system and an acquisition strategy allocation mechanism are additionally arranged in the network data acquisition system, namely, a preamble page of the target data content of the target website is acquired at regular time through a preamble network data acquisition example, and a URL (uniform resource locator) acquired by the preamble network data acquisition example is allocated to other network data acquisition examples to continue to execute subsequent acquisition operation.

Based on this, how to accurately identify the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy allocation mechanism thereof is a problem which needs to be solved urgently at present.

Disclosure of Invention

The embodiment of the application provides a network data acquisition identification method and device, electronic equipment and a computer readable storage medium, which aim to accurately identify a network data acquisition task scheduling system and data acquisition operation under an acquisition strategy allocation mechanism of the network data acquisition task scheduling system.

The embodiment of the application provides a network data acquisition and identification method, which comprises the following steps: acquiring a target encryption parameter carried by a second network request, wherein the second network request is realized based on network response data corresponding to the first network request, and the target encryption parameter is obtained after encryption processing is performed on a first network behavior background identifier corresponding to the first network request; decrypting the target encryption parameter to obtain a first network behavior background identifier; acquiring a second network behavior background identifier corresponding to the second network request; and comparing the first network behavior background identification with the second network behavior background identification, and determining whether the access subject corresponding to the second network request is a network data acquisition subject or not based on the comparison result.

Optionally, the first network behavior context identifier includes: a first network requests corresponding user context information; the second network behavior context identification comprises: the second network requests the corresponding user context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

Optionally, the user context information includes at least one of the following: user identification information; network operating environment information; comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, comprising: comparing the user identification information corresponding to the first network request with the user identification information corresponding to the second network request; and/or comparing the network operation environment information corresponding to the first network request with the network operation environment information corresponding to the second network request. Optionally, the user identification information includes: user identification information, or login session identification information for anonymous users. The network operating environment information includes at least one of: network device identification information; network device IP address.

Optionally, the first network behavior context identifier includes: the first network requests corresponding application context information; the second network behavior context identification comprises: the second network requests corresponding application context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is a network data acquisition subject based on the comparison result, including: comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

Optionally, the first network behavior context identifier includes: the user context information corresponding to the first network request and the application context information corresponding to the first network request; the second network behavior context identification comprises: the user context information corresponding to the second network request and the application context information corresponding to the second network request; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, and/or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

Optionally, the application context information includes at least one of the following: applying channel information; and applying the interface information. Optionally, the method further includes: the second network request is intercepted. Optionally, the obtaining of the target encryption parameter carried by the second network request includes: acquiring a target encryption parameter carried by a second network request aiming at target network data; the second network request is realized based on the network response data corresponding to the first network request, and comprises the following steps: the second network request is implemented based on the access policy data for the target network data contained in the network response data corresponding to the first network request.

Optionally, the method further includes: obtaining network response data in response to the first network request, the network response data including initial access policy data for the target network data; acquiring a first network behavior background identifier corresponding to a first network request; encrypting a first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter; obtaining target access policy data aiming at the target network data according to the initial access policy data and the target encryption parameter, and providing the target access policy data to an access subject corresponding to the first network request; or, the initial access policy data and the target encryption parameter are provided to the access subject corresponding to the first network request, so that the access subject obtains the target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

Optionally, the access policy data for the target network data includes: and the URL corresponding to the target network data. Optionally, the target encryption parameter is included in a parameter of the URL. Optionally, the encrypting the first network behavior context identifier corresponding to the first network request includes: and encrypting the identification information of the target network data and the first network behavior background identification. The initial access strategy data aiming at the target network data is an initial URL corresponding to the target network data, and the target access strategy data aiming at the target network data is a target URL obtained after the identification information of the target network data contained in the initial URL is replaced by a target encryption parameter; providing the target access policy data to an access subject corresponding to the first network request, including: and providing the target URL to an access subject corresponding to the first network request.

Optionally, providing the initial access policy data and the target encryption parameter to the access agent corresponding to the first network request includes: and sending the data structure file containing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request. The target encryption parameter is included in a request header or a request body of the second network request.

Optionally, the first network request includes: an access main body corresponding to the first network request shares target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: a network behavior background identifier corresponding to the target sharing subject; the method for encrypting the first network behavior background identifier corresponding to the first network request comprises the following steps: and encrypting the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject.

Optionally, the encrypting the first network behavior context identifier corresponding to the first network request includes: encrypting the first network behavior background identification by using a target encryption key in the multiple sets of keys, wherein the second network request comprises a target encryption key corresponding to the target key identification; correspondingly, the decryption of the target encryption parameter includes: and decrypting the target encryption parameter by using the target decryption key corresponding to the target key identifier. Optionally, the encrypting the first network behavior context identifier corresponding to the first network request includes: and encrypting the first network behavior background identification corresponding to the first network request based on a symmetric key cryptosystem. The first network behavior context identification comprises: time stamp information. Optionally, the first network request is a network access request for a merchant list page, and the second network request is a network access request for a merchant detail page.

An embodiment of the present application further provides a network data acquiring and identifying device, including: a target encryption parameter obtaining unit, configured to obtain a target encryption parameter carried by a second network request, where the second network request is implemented based on network response data corresponding to a first network request, and the target encryption parameter is a parameter obtained after encrypting a first network behavior background identifier corresponding to the first network request; the target encryption parameter decryption unit is used for decrypting the target encryption parameter to obtain a first network behavior background identifier; a second network behavior background identifier acquiring unit, configured to acquire a second network behavior background identifier corresponding to the second network request; and the network behavior background identifier comparison unit is used for comparing the first network behavior background identifier with the second network behavior background identifier and determining whether the access subject corresponding to the second network request is a network data acquisition subject or not based on the comparison result.

Optionally, the user context information includes at least one of the following: user identification information; network operating environment information; comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, comprising: comparing the user identification information corresponding to the first network request with the user identification information corresponding to the second network request; and/or comparing the network operation environment information corresponding to the first network request with the network operation environment information corresponding to the second network request. Optionally, the user identification information includes: user identification information, or login session identification information for anonymous users. Optionally, the network operating environment information includes at least one of the following: network device identification information; network device IP address.

Optionally, the first network behavior context identifier includes: a first network requests corresponding application context information; the second network behavior context identification comprises: the second network requests corresponding application context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is a network data acquisition subject based on the comparison result, including: comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that an access subject corresponding to the second network request is a network data acquisition subject.

Optionally, the first network behavior context identifier includes: the user context information corresponding to the first network request and the application context information corresponding to the first network request; the second network behavior context identification comprises: the user context information corresponding to the second network request and the application context information corresponding to the second network request; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is a network data acquisition subject based on the comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, and/or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject. Optionally, the application context information includes at least one of the following: applying channel information; application interface information. Optionally, the method further includes: and the second network request intercepting unit is used for intercepting the second network request.

Optionally, the obtaining of the target encryption parameter carried by the second network request includes: acquiring a target encryption parameter carried by a second network request aiming at target network data; the second network request is realized based on the network response data corresponding to the first network request, and comprises the following steps: the second network request is implemented based on access policy data for the target network data contained in the network response data corresponding to the first network request.

Optionally, the method further includes: a network response data acquisition unit configured to acquire network response data in response to the first network request, the network response data including initial access policy data for the target network data; a first network behavior background identifier obtaining unit, configured to obtain a first network behavior background identifier corresponding to the first network request; the encryption processing unit is used for encrypting the first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter; the data providing unit is used for obtaining target access policy data aiming at the target network data according to the initial access policy data and the target encryption parameter and providing the target access policy data to an access subject corresponding to the first network request; or, providing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request, so that the access subject obtains target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

Optionally, the access policy data for the target network data includes: and the URL corresponding to the target network data. Optionally, the target encryption parameter is included in a parameter of the URL.

Optionally, the encrypting the first network behavior context identifier corresponding to the first network request includes: and encrypting the identification information of the target network data and the first network behavior background identification.

Optionally, the initial access policy data for the target network data is an initial URL corresponding to the target network data, and the target access policy data for the target network data is a target URL obtained after replacing the identification information of the target network data included in the initial URL with the target encryption parameter; the providing the target access policy data to the access subject corresponding to the first network request includes: and providing the target URL to an access subject corresponding to the first network request. Optionally, the providing the initial access policy data and the target encryption parameter to the access subject corresponding to the first network request includes: and sending a data structure file containing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request.

Optionally, the target encryption parameter is included in a request header or a request body of the second network request.

Optionally, the first network request includes: an access main body corresponding to a first network request shares the target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: and the target shares the network behavior background identification corresponding to the main body. The encrypting the first network behavior background identifier corresponding to the first network request includes: and encrypting the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject.

Optionally, the encrypting the first network behavior context identifier corresponding to the first network request includes: encrypting the first network behavior background identifier by using a target encryption key in a plurality of sets of keys, wherein the second network request comprises a target key identifier corresponding to the target encryption key; correspondingly, the decrypting the target encryption parameter includes: and decrypting the target encryption parameter by using a target decryption key corresponding to the target key identifier.

Optionally, the encrypting the first network behavior context identifier corresponding to the first network request includes: and encrypting the first network behavior background identification corresponding to the first network request based on a symmetric key cryptosystem. The first network behavior context identification comprises: time stamp information.

Optionally, the first network request is a network access request for a merchant list page, and the second network request is a network access request for a merchant detail page.

The embodiment of the application also provides an electronic device, which comprises a processor and a memory; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method.

Embodiments of the present application also provide a computer-readable storage medium having one or more computer instructions stored thereon, which are executed by a processor to implement the above-mentioned method.

Compared with the prior art, the embodiment of the application has the following advantages:

in the network data acquisition and identification method provided by the embodiment of the application, after a target encryption parameter carried by a second network request is acquired, the target encryption parameter is decrypted to obtain a first network behavior background identifier, a second network behavior background identifier corresponding to the second network request is acquired, the first network behavior background identifier is compared with the second network behavior background identifier, whether an access subject corresponding to the second network request is a network data acquisition subject is determined based on a comparison result, the second network request is realized based on network response data corresponding to the first network request, and the target encryption parameter is a parameter obtained after encryption processing is performed on the first network behavior background identifier corresponding to the first network request. Since the second network request is implemented based on the network response data corresponding to the first network request, it may be determined whether the second network request is from the network data obtaining subject based on whether the first network request and the second network request correspond to the same network behavior context identifier, for example, if the second network request is a network request from a normal user, the first network request and the second network request should correspond to the same network behavior context identifier, and if the first network request and the second network request correspond to different network behavior context identifiers, it indicates that the network data obtaining operation implemented based on the network data obtaining task scheduling system is between the first network request and the second network request, that is, the access subject corresponding to the second network request is the network data obtaining subject. By using the method, the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy distribution mechanism can be accurately identified.

Drawings

Fig. 1 is a flowchart of a network data acquisition and identification method according to a first embodiment of the present application;

FIG. 2 is a schematic view of a scenario provided by a first embodiment of the present application;

fig. 3 is a block diagram of elements of a network data acquisition and identification device according to a second embodiment of the present application;

fig. 4 is a schematic logical structure diagram of an electronic device according to a third embodiment of the present application.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.

Aiming at a network data acquisition and identification scene, in order to accurately identify network data acquisition behaviors, the application provides a network data acquisition and identification method, a network data acquisition and identification device corresponding to the method, electronic equipment and a computer readable storage medium. The following embodiments are provided to describe the method, apparatus, electronic device, and computer-readable storage medium in detail.

A first embodiment of the present application provides a network data acquisition and identification method, an application body of the method may be a computing device application for identifying a network data acquisition behavior, the computing device application may be run in a network server (for example, a network platform server in an online shopping scenario or a life service type network application scenario) or run in a network middleware (for example, a terminal device dedicated to identifying a network data acquisition behavior) independently deployed between the network server and a client, fig. 1 is a flowchart of a network data acquisition and identification method provided in the first embodiment of the present application, fig. 2 is a schematic view of a scenario provided in the first embodiment of the present application, and the network data acquisition and identification method provided in this embodiment is described in detail below with reference to fig. 1 and fig. 2. The following description refers to embodiments for illustrating the principles of the methods and is not meant to be limiting in actual use.

As shown in fig. 1, the network data acquiring and identifying method provided in this embodiment includes the following steps:

s101, acquiring a target encryption parameter carried by the second network request.

The second network request is implemented based on the network response data corresponding to the first network request, that is, for the network access user, the first network request and the second network request have a network access hierarchy relationship, the second network request is implemented based on the network response data corresponding to the first network request, the first network request is a preamble request of the second network request, and the page accessed by the first network request and the page accessed by the second network request are continuous pages. As shown in fig. 2, in an online shopping scenario or a life service type network application scenario, the first network request may be a network access request of a user for a merchant list page, and the second network request may be a network access request of a user for a merchant detail page of a merchant in the merchant list page.

The target encryption parameter is a parameter obtained after encrypting a first network behavior background identifier corresponding to the first network request, that is, the target encryption parameter can be obtained by using the first network behavior background identifier as encrypted data and encrypting the encrypted data by using a predetermined encryption function, so that the encryption processing of the first network behavior is performed to avoid that a network data acquisition main body autonomously generates an access parameter, thereby avoiding the network data acquisition and identification method of the embodiment.

The network behavior background identifier is used for identifying background information corresponding to the network access behavior, and due to differences of factors such as a network access subject, a network environment, an application link and the like, each network access behavior corresponds to the network behavior background identifier, which may be user context information corresponding to the network access behavior, may also be application context information corresponding to the network access behavior, and may also include user context information and application context information corresponding to the same network access behavior.

The user context information is used for identifying a network access subject, and for access requests from different users, the user context information corresponds to different user context information, in this embodiment, the user context information may be at least one of user identification information and network operation environment information, the user identification information may be user identification information (e.g., a user ID) of a registered user, or may be login session identification information allocated for an anonymous user (e.g., in order to identify a user identity, a website allocates a login session ID to an anonymous user lacking a user ID and lacking an associated login session in advance, and stores the login session ID in a Cookie, which indicates state information transferred between a network server and a client and may be used for tracking the user); the network operating environment information is used to identify a network operating environment corresponding to the network access behavior, and may be at least one of network device identification information (e.g., a network device ID) and a network device IP address.

The application context information is used to identify an application link corresponding to the network access behavior, and may be at least one of application channel information corresponding to the network request (for example, the network requests from different network platforms correspond to different application channel information) and application interface information (for example, the network requests from different application interfaces of the same network platform correspond to different application interface information).

In this embodiment, the obtaining of the target encryption parameter carried by the second network request may refer to: acquiring a target encryption parameter carried by a second network request aiming at target network data; correspondingly, the second network request is implemented based on the network response data corresponding to the first network request, and may refer to: the second network request is implemented based on the access policy data for the target network data included in the network response data corresponding to the first network request, in this embodiment, the target encryption parameter may be included in the access policy data, the access policy data may be a URL (Uniform Resource Locator) corresponding to the target network data, for example, a merchant list page (first network request) of the online shopping platform is accessed by the user, in the content of the merchant list to be returned by the network platform server, the merchant detail page of each merchant corresponds to an initial URL, after adding the target encryption parameter in each initial URL or replacing part of parameters in the initial URL with the target encryption parameter, a target page corresponding to the merchant detail page is obtained, and the target URL is returned to the user as the network response data of the first network request, when the user accesses the merchant detail page (second network request) of the target merchant based on the target URL (clicking a link of the target URL), the merchant page of the target merchant is the target network data, and the target URL includes the target encryption parameter.

It should be noted that the network response data corresponding to the first network request may also be a data structure file with a predetermined format, and correspondingly, the access policy data may also be included in a data structure file with a predetermined format returned by the network server, for example, the access policy data may be included in a JSON file returned by the network server, and the client may assemble a URL corresponding to the target network data based on the JSON file, add a target encryption parameter to the URL, or replace a part of the parameter in the URL with a target encryption parameter, and then obtain a target URL corresponding to the target network data, and access the target network data based on the target URL (the second network request).

It should be noted that, in addition to the parameter of the URL, the target encryption parameter may also be included in a request header (header) or a request body (body) of the second network request, and it is only necessary to implement that the second network request carries the target encryption parameter, which is not limited herein.

Corresponding to the above, before executing step S101, the application main body of the present embodiment further needs to perform the following operations:

firstly, network response data responding to a first network request is obtained, the network response data comprises initial access policy data aiming at target network data, the initial access policy data can be an initial URL corresponding to the target network data, for example, a merchant list page (first network request) of a network shopping platform accessed by a user, and in merchant list contents to be returned by a network platform server, the merchant list page of each merchant corresponds to the initial URL;

secondly, acquiring a first network behavior background identifier corresponding to the first network request, for example, acquiring a user ID, a login session ID, a network device IP address, application channel information, and the like corresponding to the first network request;

then, encrypting a first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter;

finally, according to the initial access policy data and the target encryption parameter, target access policy data for the target network data is obtained, and the target access policy data is provided to the access subject corresponding to the first network request, for example, the target access policy data may be a target URL obtained after the implementation subject replaces identification information of the target network data included in the initial URL with the target encryption parameter, or a target URL obtained after the implementation subject adds the target encryption parameter on the basis of the original parameter in the initial URL, and correspondingly, the providing of the target access policy data to the access subject corresponding to the first network request may refer to the implementation subject providing the target URL to the access subject corresponding to the first network request. The identification information of the target network data is used to identify the target network data in the content addressing process of the network platform, and may be one of the parameters in the initial URL, for example, a delivery parameter (e.g., a merchant internal ID) preset by the network platform for an resident merchant, for specifying the merchant between systems inside the network platform.

In this embodiment, in addition to obtaining the target access policy data for the target network data according to the initial access policy data and the target encryption parameter and providing the target access policy data to the access subject corresponding to the first network request, the initial access policy data and the target encryption parameter may be provided to the access subject corresponding to the first network request, so that the access subject obtains the target access policy data for the target network data based on the initial access policy data and the target encryption parameter, for example, the target access policy data may be a target URL obtained after the identification information of the target network data contained in the initial URL is replaced by the target encryption parameter for the client (the access subject corresponding to the first network request) or a target URL obtained after the target encryption parameter is added on the basis of the original parameter in the initial URL for the client (the access subject corresponding to the first network request). The providing the initial access policy data and the target encryption parameter to the access subject corresponding to the first network request may also refer to: and sending a data structure file containing initial access strategy data and target encryption parameters to an access main body corresponding to the first network request, so that the access main body assembles the URL corresponding to the target network data based on the data structure file, adds the target encryption parameters to the URL, obtains the target URL, and accesses the target network data based on the target URL.

In this embodiment, in order to increase the applicability (for example, to make the encryption process adapt to different application lines of the network platform) and the security (for example, to avoid the influence on the encryption and decryption processes due to factors such as application system upgrade and data update of the network platform), the encryption process may be preferentially implemented based on multiple sets of keys preset by the network platform, for example, a target encryption key in the multiple sets of keys is used to encrypt a first network behavior background identifier, each set of key corresponds to a key identifier (for example, a key ID corresponding to each set of key, or a hash value obtained by performing hash calculation on the key), in this case, a second network request includes the target key identifier corresponding to the target encryption key, for example, the target key identifier is used as a new parameter in an initial URL or returned to the client in another manner, and the target key identifier may also be a part of encrypted data. Subsequent decryption processes may be implemented based on the target key identification.

It should be noted that, the encryption process may be implemented based on a symmetric key cryptosystem (the encryption key and the decryption key use the same cryptosystem) or a public key cryptosystem (different encryption key and decryption key are used), in this embodiment, in order to improve convenience of the encryption and decryption process, the first network behavior background identifier corresponding to the first network request is preferably encrypted based on the symmetric key cryptosystem, for example, for the same network platform, convenience of the encryption and decryption process may be improved by using the symmetric key cryptosystem for encryption and decryption.

In this embodiment, the first network behavior context identifier may further include timestamp information corresponding to the first network request, and the timestamp information is used as a part of the encrypted data, so that the validity period of the encryption result may be limited in a time dimension, and replay attack may be prevented.

S102, the target encryption parameters are decrypted to obtain a first network behavior background identifier.

After the target encryption parameter carried by the second network request is obtained in the above step, the step is used for performing decryption processing on the target encryption parameter, so as to obtain a first network behavior background identifier corresponding to the first network request.

Corresponding to the above digital encryption based on multiple sets of keys, in this embodiment, decrypting the target encryption parameter may specifically refer to: and decrypting the target encryption parameter by using the target decryption key corresponding to the target key identifier.

S103, acquiring a second network behavior background identifier corresponding to the second network request.

After the target encryption parameter is decrypted and the first network behavior background identifier is obtained in the above steps, this step is used to obtain the second network behavior background identifier corresponding to the second network request, for example, obtain one or more of information such as a user ID, a login session ID, a network device IP address, and application channel information corresponding to the second network request.

It should be noted that, in this embodiment, the implementation sequence of step S102 and step S103 is not limited, that is, after obtaining the second network behavior context identifier corresponding to the second network request, a decryption operation may be performed on the target encryption parameter carried by the second network request, and the first network behavior context identifier is obtained. If step S103 is executed after step S102 is executed, the second network behavior context identifier corresponding to the second network request may be obtained in a manner suitable for the decryption result, that is, based on the first network behavior context identifier obtained by decryption in step S102, the second network behavior context identifier of the same category as the first network behavior context identifier may be selectively obtained.

And S104, comparing the first network behavior background mark with the second network behavior background mark, and determining whether an access subject corresponding to the second network request is a network data acquisition subject or not based on the comparison result.

On the basis of obtaining the first network behavior background identifier and the second network behavior background identifier in the above steps, the step is used for comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result.

The purpose of comparing the first network behavior background identifier with the second network behavior background identifier is to determine whether the two network requests correspond to the same network access subject, or correspond to the same network operating environment, or correspond to the same application link. For a normal network request, the probability of cross-user access, or cross-network operating environment access, or cross-application link access between a first network request and a second network request having a network access hierarchical relationship is low, and therefore, whether an access subject corresponding to the second network request is a network data acquisition subject can be determined based on a preconfigured network data acquisition identification dimension and the result of the comparison.

In this embodiment, the above process can be implemented as follows:

the method I comprises the following steps: the first network behavior background identifier is user context information corresponding to the first network request; the second network behavior background mark is user context information corresponding to the second network request; the comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is the network data acquisition subject based on the comparison result specifically means: comparing user context information corresponding to the first network request with user context information corresponding to the second network request so as to determine whether user contexts corresponding to the two network requests are matched, for example, comparing user identification information (user identification information of a registered user or login session identification information distributed for an anonymous user) corresponding to the first network request with user identification information corresponding to the second network request based on a user context comparison dimension pre-configured for identification and interception requirements of network data acquisition behaviors of a network platform, or comparing network operation environment information (at least one of a network device ID and a network device IP address) corresponding to the first network request with network operation environment information corresponding to the second network request, or comparing the user identification information and the network operation environment information corresponding to the first network request with the user identification information and the network operation environment information corresponding to the second network request, thereby determining whether the two network requests correspond to the same access subject or the same network operation environment; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject. By the method, the characteristic that the network data acquisition task system performs page access in a mode of switching user contexts among a plurality of webpages of the network platform can be resisted, for example, the characteristic that the network data acquisition task system frequently changes users to acquire webpage data can be resisted by comparing user IDs of two network requests; by comparing the login session identification information distributed by the anonymous user, the characteristic that the network data acquisition system frequently changes the anonymous user to acquire the webpage can be resisted; by comparing network operation environment information such as network equipment ID, network equipment IP address and the like, the characteristic that the network operation environment is frequently changed by the network data acquisition task system can be resisted.

The second method comprises the following steps: the first network behavior background identifier is application context information corresponding to the first network request, the second network behavior background identifier is application context information corresponding to the second network request, the first network behavior background identifier is compared with the second network behavior background identifier, and whether an access subject corresponding to the second network request is a network data acquisition subject is determined based on a comparison result, specifically, the method includes: comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that an access subject corresponding to the second network request is a network data acquisition subject. By the method, the characteristic that the network data acquisition task system frequently changes an application link (an application channel or an application interface) to acquire the webpage can be resisted.

The third method comprises the following steps: the process of comparing the first network behavior background identifier with the second network behavior background identifier and determining whether the access subject corresponding to the second network request is the network data acquisition subject based on the comparison result may also be implemented based on a comparison dimension pre-configured for the identification and interception requirements of the network data acquisition behavior of the network platform, and the process may specifically refer to: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, and the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

It should be noted that, in the data sharing operation of a normal user, there are also situations that the first network request and the second network request correspond to different network access subjects, different network operation environments, and different application links, for example, the user a shares link information corresponding to a merchant detail page to the user B, and the user B accesses the merchant detail page based on the link information, so in order to avoid the data sharing operation of the normal user and the data acquisition operation of the shared user that are mistakenly recognized as a data acquisition operation of a network data acquisition subject, the first network request may also refer to a sharing request that an access subject corresponding to the first network request shares target network data to a target sharing subject, where the target sharing subject is a shared user. In this scenario, the encrypting the first network behavior context identifier corresponding to the first network request means: the network behavior background identifier corresponding to the target sharing subject is encrypted, for example, information such as a user ID corresponding to the user B and a device ID of a terminal device used by the user B is encrypted. When the first network behavior background identifier is compared with the second network behavior background identifier, if the user is a normal user, the network behavior background identifier corresponding to the target sharing subject should be consistent with the network behavior background identifier corresponding to the second network request. In this embodiment, the encryption process may further be: the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject are encrypted (for example, the user ID of the user a, the user ID of the user B, and the device ID of the terminal device used by the user B are encrypted), and the user identification information of the access subject corresponding to the first network request is used as a part of encrypted data, so that the source tracing of the sharing behavior can be realized.

In this embodiment, the encrypting the first network behavior context identifier corresponding to the first network request in step S101 may further refer to: the identification information (such as a merchant internal ID) of the target network data and the first network behavior background identification are encrypted, so that the identification information of the target network data is used as a part of encrypted data, and the purpose is to set a checking link for a response process of the target network data.

In this embodiment, after determining that the access subject corresponding to the second network request is the network data acquisition subject through the above steps, intercepting or performing early warning processing on the second network request is further required.

In the method for acquiring and identifying network data provided in this embodiment, after a target encryption parameter carried by a second network request is acquired, the target encryption parameter is decrypted to obtain a first network behavior background identifier (one or more of user context information such as a user ID, a login session ID, a network device IP address, and application context information such as application channel information and application interface information corresponding to the first network request), and obtain a second network behavior background identifier corresponding to the second network request (one or more of user context information such as a user ID, a login session ID, a network device IP address, and application context information such as application channel information and application interface information corresponding to the second network request), compare the first network behavior background identifier with the second network behavior background identifier, and determine whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, where the second network request is implemented based on network response data corresponding to the first network request, and the target encryption parameter is obtained after the first network request is processed on the first network behavior background identifier. Because the second network request is implemented based on the network response data corresponding to the first network request, it may be determined whether the second network request is from the network data acquisition subject based on whether the first network request and the second network request correspond to the same network behavior context identifier, for example, if the second network request is a network request from a normal user, the first network request and the second network request correspond to the same network behavior context identifier, that is, the first network request and the second network request correspond to the same user context information such as the user ID, the login session ID, the network device IP address, and the like, and the application context information such as the application channel information, the application interface information, and the like; if the first network request and the second network request correspond to different network behavior background identifications, for example, the first network request and the second network request correspond to different user IDs or login session IDs, the first network request and the second network request are cross-user network requests; the first network request and the second network request correspond to different network equipment IDs or network equipment IP addresses, and the first network request and the second network request are network requests of a cross-network operation environment; the first network request and the second network request correspond to different application channel information or application interface information, and the first network request and the second network request are network requests crossing application links; however, for the first network request and the second network request having a network access hierarchical relationship, the possibility of the cross-user, cross-network operating environment and cross-application link is low, and for the network data acquisition task scheduling system and the data acquisition operation under the acquisition policy allocation mechanism thereof, in order to bypass the limitation of the access amount of a single IP or a single account in the identification and interception technology for the network data acquisition behavior and achieve the purpose of acquiring a large amount of network data, when performing acquisition policy allocation for the first network request and the second network request having a network access hierarchical relationship, it is indispensable to implement the network data acquisition operation implemented by the cross-user, cross-network operating environment and cross-application link, so if the first network request and the second network request correspond to different network behavior context identifiers, it is indicated that the network data acquisition operation implemented based on the network data acquisition task scheduling system is performed between the first network request and the second network request, that the access subject corresponding to the second network request is the network data acquisition subject. By using the method, the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy distribution mechanism can be accurately identified.

In order to avoid the network data acquisition and identification method provided by this embodiment, the network data acquisition system has to use the same account to execute the first network request and the second network request having the network access hierarchical relationship under the same network operating environment and the same application link, so as to acquire consecutive pages, in this case, there are two possible results as follows: 1. the request times of a single account are increased, so that each IP and each account have obvious intensive operation, and the network data acquisition behavior is easy to be identified and intercepted by the existing identification and interception technology aiming at the network data acquisition behavior; II, secondly, the method comprises the following steps: the total request number of the network data acquisition behaviors is increased, and the data acquisition cost is increased.

For example, the network data acquisition system wants to acquire a first page of a merchant list page and a merchant detail page corresponding to a second page in an online shopping platform, each merchant list page includes links of 20 merchant detail pages, and under the existing network data acquisition task scheduling system and an acquisition policy distribution mechanism thereof, the acquisition of the merchant detail pages can be completed only by allocating 42 (2 +20 +2, wherein 2 network data acquisition instances acquire the merchant list page) network data acquisition instances and executing 42 page access requests; if the network data acquisition and identification method provided by this embodiment is adopted to protect the online shopping platform, the network data acquisition system needs to improve its task system to avoid this scheme, for example, the network data acquisition system needs to allocate 20 network data acquisition instances to attack the merchant detail page corresponding to the top page of the merchant list page, each network data acquisition instance needs to acquire the top page and the corresponding merchant detail page respectively, 40 page access requests in total need to be executed, and 20 network data acquisition instances need to attack the merchant detail page corresponding to the second page of the merchant list page, each network data acquisition instance needs to acquire the top page, the second page, and the corresponding merchant detail page of the merchant list page, 20 × 3 page access requests need to be executed, that is, the network data acquisition system needs to execute 100 page access requests in total, and its data acquisition cost is significantly raised; in order to reduce the data acquisition cost and avoid the network data acquisition and identification method provided by this embodiment, the network data acquisition system may use a single network data acquisition instance to attack all targets, that is, the same network data acquisition subject is used to execute 42 page access requests to acquire all merchant detail pages corresponding to the first page and the second page of the merchant list page, and the total request number is consistent with the total request number under the existing network data acquisition task scheduling system and its acquisition policy allocation mechanism, however, the intensive operation of a single account is easily identified and intercepted by the existing identification and interception technology for network data acquisition behavior based on IP dimension and account dimension.

The second embodiment of the present application further provides a network data acquisition and identification device, which is basically similar to the method embodiment and therefore is relatively simple to describe, and reference may be made to the corresponding description of the method embodiment for details of related technical features, and the following description of the device embodiment is merely illustrative. Referring to fig. 3, to understand the embodiment, fig. 3 is a block diagram of a unit of the apparatus provided in the embodiment, and as shown in fig. 3, the apparatus provided in the embodiment includes: a target encryption parameter obtaining unit 201, configured to obtain a target encryption parameter carried by a second network request, where the second network request is implemented based on network response data corresponding to a first network request, and the target encryption parameter is obtained after a first network behavior background identifier corresponding to the first network request is encrypted; the target encryption parameter decryption unit 202 is configured to decrypt the target encryption parameter to obtain a first network behavior background identifier; a second network behavior background identifier obtaining unit 203, configured to obtain a second network behavior background identifier corresponding to the second network request; the network behavior background identifier comparing unit 204 is configured to compare the first network behavior background identifier with the second network behavior background identifier, and determine whether the access subject corresponding to the second network request is a network data obtaining subject based on a comparison result.

The first network behavior context identification comprises: a first network requests corresponding user context information; the second network behavior context identification comprises: the second network requests the corresponding user context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

The user context information includes at least one of: user identification information; network operating environment information; comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, comprising: comparing the user identification information corresponding to the first network request with the user identification information corresponding to the second network request; and/or comparing the network operation environment information corresponding to the first network request with the network operation environment information corresponding to the second network request. The user identification information includes: user identification information, or login session identification information for anonymous users. The network operating environment information includes at least one of: network device identification information; network device IP address.

The first network behavior context identification comprises: the first network requests corresponding application context information; the second network behavior context identification comprises: the second network requests corresponding application context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is a network data acquisition subject based on the comparison result, including: comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that an access subject corresponding to the second network request is a network data acquisition subject.

The first network behavior context identification comprises: the user context information corresponding to the first network request and the application context information corresponding to the first network request; the second network behavior context identification comprises: the user context information corresponding to the second network request and the application context information corresponding to the second network request; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, and/or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is the network data acquisition subject.

The application context information includes at least one of: applying channel information; application interface information. Further comprising: and the second network request intercepting unit is used for intercepting the second network request. Acquiring a target encryption parameter carried by the second network request, including: acquiring a target encryption parameter carried by a second network request aiming at target network data; the second network request is realized based on the network response data corresponding to the first network request, and comprises the following steps: the second network request is implemented based on the access policy data for the target network data contained in the network response data corresponding to the first network request.

Further comprising: a network response data acquisition unit configured to acquire network response data in response to the first network request, the network response data including initial access policy data for the target network data; a first network behavior background identifier obtaining unit, configured to obtain a first network behavior background identifier corresponding to the first network request; the encryption processing unit is used for encrypting the first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter; the data providing unit is used for obtaining target access strategy data aiming at the target network data according to the initial access strategy data and the target encryption parameters and providing the target access strategy data to an access subject corresponding to the first network request; or, the initial access policy data and the target encryption parameter are provided to the access subject corresponding to the first network request, so that the access subject obtains the target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

The access policy data for the target network data includes: and URL corresponding to the target network data. The target encryption parameter is included in the parameters of the URL. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the identification information of the target network data and the first network behavior background identification. The initial access strategy data aiming at the target network data is an initial URL corresponding to the target network data, and the target access strategy data aiming at the target network data is a target URL obtained after the identification information of the target network data contained in the initial URL is replaced by a target encryption parameter; providing the target access policy data to an access subject corresponding to the first network request, including: and providing the target URL to the access subject corresponding to the first network request.

Providing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request, including: and sending the data structure file containing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request. The target encryption parameter is included in a request header or a request body of the second network request. The first network request includes: an access main body corresponding to the first network request shares target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: and the target shares the network behavior background mark corresponding to the main body. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject.

The method for encrypting the first network behavior background identifier corresponding to the first network request comprises the following steps: encrypting the first network behavior background identification by using a target encryption key in the multiple sets of keys, wherein the second network request comprises a target encryption key corresponding to the target key identification; correspondingly, decrypting the target encryption parameter comprises the following steps: and decrypting the target encryption parameter by using the target decryption key corresponding to the target key identifier.

The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the first network behavior background identification corresponding to the first network request based on a symmetric key cryptosystem. The first network behavior context identification comprises: time stamp information. The first network request is a network access request for a merchant listing page and the second network request is a network access request for a merchant details page.

The network data acquisition and identification device provided in this embodiment decrypts a target encryption parameter after acquiring the target encryption parameter carried by a second network request, obtains a first network behavior background identifier (one or more of user context information such as a user ID, a login session ID, a network device IP address, and application context information such as application channel information and application interface information corresponding to the first network request), acquires a second network behavior background identifier corresponding to the second network request (one or more of user context information such as a user ID, a login session ID, a network device IP address, and application context information such as application channel information and application interface information corresponding to the second network request), compares the first network behavior background identifier with the second network behavior background identifier, and determines whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, where the second network request is implemented based on network response data corresponding to the first network request, and the target encryption parameter is obtained after processing the first network behavior background identifier corresponding to the first network request. By using the device, accurate identification can be carried out on the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy distribution mechanism.

In the embodiments described above, a network data acquisition and identification method and a network data acquisition and identification device are provided, and in addition, a third embodiment of the present application also provides an electronic device, which is basically similar to the method embodiment and therefore is relatively simple to describe, and reference may be made to the corresponding description of the method embodiment for details of relevant technical features, and the following description of the embodiment of the electronic device is only illustrative. The embodiment of the electronic equipment is as follows: please refer to fig. 4 for understanding the present embodiment, fig. 4 is a schematic view of an electronic device provided in the present embodiment. As shown in fig. 4, the electronic device provided in this embodiment includes: a processor 301 and a memory 302; the memory 302 is used for storing computer instructions of the network data acquisition identification method, and when the computer instructions are read and executed by the processor 301, the computer instructions perform the following operations: acquiring a target encryption parameter carried by a second network request, wherein the second network request is realized based on network response data corresponding to the first network request, and the target encryption parameter is obtained after encryption processing is performed on a first network behavior background identifier corresponding to the first network request; decrypting the target encryption parameter to obtain a first network behavior background identifier; acquiring a second network behavior background identifier corresponding to the second network request; and comparing the first network behavior background identification with the second network behavior background identification, and determining whether the access subject corresponding to the second network request is a network data acquisition subject or not based on the comparison result.

The first network behavior context identification comprises: a first network requests corresponding user context information; the second network behavior context identification comprises: the second network requests the corresponding user context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is a network data acquisition subject based on the comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

The first network behavior context identification comprises: the user context information corresponding to the first network request and the application context information corresponding to the first network request; the second network behavior context identification comprises: the user context information corresponding to the second network request and the application context information corresponding to the second network request; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, and/or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject. The application context information includes at least one of: applying channel information; application interface information. Further comprising: a second network request is intercepted. Acquiring a target encryption parameter carried by the second network request, including: acquiring a target encryption parameter carried by a second network request aiming at target network data; the second network request is realized based on the network response data corresponding to the first network request, and comprises the following steps: the second network request is implemented based on the access policy data for the target network data contained in the network response data corresponding to the first network request. Further comprising: obtaining network response data in response to the first network request, the network response data including initial access policy data for the target network data; acquiring a first network behavior background identifier corresponding to a first network request; encrypting a first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter; obtaining target access policy data aiming at the target network data according to the initial access policy data and the target encryption parameter, and providing the target access policy data to an access subject corresponding to the first network request; or, the initial access policy data and the target encryption parameter are provided to the access subject corresponding to the first network request, so that the access subject obtains the target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

The access policy data for the target network data includes: and the URL corresponding to the target network data. The target encryption parameter is included in the parameters of the URL. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the identification information of the target network data and the first network behavior background identification. The initial access strategy data aiming at the target network data is an initial URL corresponding to the target network data, and the target access strategy data aiming at the target network data is a target URL obtained after the identification information of the target network data contained in the initial URL is replaced by a target encryption parameter; providing the target access policy data to an access subject corresponding to the first network request, including: and providing the target URL to an access subject corresponding to the first network request. Providing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request, including: and sending the data structure file containing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request. The target encryption parameter is included in a request header or a request body of the second network request.

The first network request includes: an access main body corresponding to the first network request shares target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: a network behavior background identifier corresponding to the target sharing subject; the method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: encrypting the first network behavior background identification by using a target encryption key in the multiple sets of keys, wherein the second network request comprises a target encryption key corresponding to the target key identification; correspondingly, the decryption of the target encryption parameter includes: and decrypting the target encryption parameter by using the target decryption key corresponding to the target key identifier. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the first network behavior background identification corresponding to the first network request based on a symmetric key cryptosystem. The first network behavior context identification comprises: time stamp information. The first network request is a network access request for a merchant list page, and the second network request is a network access request for a merchant detail page.

In the electronic device provided in this embodiment, after a target encryption parameter carried by a second network request is obtained, the target encryption parameter is decrypted, a first network behavior background identifier (one or more of user context information such as a user ID, a login session ID, a network device IP address, and application context information such as application channel information and application interface information corresponding to the first network request) is obtained, a second network behavior background identifier corresponding to the second network request (one or more of user context information such as a user ID, a login session ID, a network device IP address, and application context information such as application channel information and application interface information corresponding to the second network request) is obtained, the first network behavior background identifier is compared with the second network behavior background identifier, and it is determined, based on a comparison result, whether an access subject corresponding to the second network request is a network data obtaining subject, the second network request is implemented based on network response data corresponding to the first network request, and the target encryption parameter is obtained after encryption of the first network request is performed on the corresponding first network behavior background identifier. By using the electronic equipment, accurate identification can be performed on the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy distribution mechanism.

In the foregoing embodiments, a network data acquisition and identification method, a network data acquisition and identification device, and an electronic device are provided. The embodiments of the computer-readable storage medium provided in the present application are described relatively simply, and for relevant portions, reference may be made to the corresponding descriptions of the above method embodiments, and the embodiments described below are merely illustrative.

The present embodiments provide a computer readable storage medium having stored thereon computer instructions that, when executed by a processor, perform the steps of: acquiring a target encryption parameter carried by a second network request, wherein the second network request is realized based on network response data corresponding to the first network request, and the target encryption parameter is obtained after encryption processing is performed on a first network behavior background identifier corresponding to the first network request; decrypting the target encryption parameter to obtain a first network behavior background identifier; acquiring a second network behavior background identifier corresponding to the second network request; and comparing the first network behavior background mark with the second network behavior background mark, and determining whether an access subject corresponding to the second network request is a network data acquisition subject or not based on the comparison result.

The first network behavior context identification comprises: the first network requests corresponding application context information; the second network behavior context identification comprises: the second network requests corresponding application context information; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, including: comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

The first network behavior context identification comprises: the user context information corresponding to the first network request and the application context information corresponding to the first network request; the second network behavior context identification comprises: the user context information corresponding to the second network request and the application context information corresponding to the second network request; comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether the access subject corresponding to the second network request is a network data acquisition subject based on the comparison result, including: comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request; and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, and/or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is the network data acquisition subject. The application context information includes at least one of: applying channel information; and applying the interface information. Further comprising: a second network request is intercepted. Acquiring a target encryption parameter carried by the second network request, including: acquiring a target encryption parameter carried by a second network request aiming at target network data; the second network request is realized based on the network response data corresponding to the first network request, and comprises the following steps: the second network request is implemented based on the access policy data for the target network data contained in the network response data corresponding to the first network request. Further comprising: obtaining network response data in response to the first network request, the network response data including initial access policy data for the target network data; acquiring a first network behavior background identifier corresponding to a first network request; encrypting a first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter; obtaining target access policy data aiming at the target network data according to the initial access policy data and the target encryption parameter, and providing the target access policy data to an access subject corresponding to the first network request; or, the initial access policy data and the target encryption parameter are provided to the access subject corresponding to the first network request, so that the access subject obtains the target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

The first network request includes: an access main body corresponding to the first network request shares target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: a network behavior background identifier corresponding to the target sharing subject; the method for encrypting the first network behavior background identifier corresponding to the first network request comprises the following steps: and encrypting the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: encrypting the first network behavior background identifier by using a target encryption key in the multiple sets of keys, wherein the second network request comprises a target encryption key corresponding to the target key identifier; correspondingly, the decryption of the target encryption parameter includes: and decrypting the target encryption parameter by using the target decryption key corresponding to the target key identifier. The method for encrypting the first network behavior background identification corresponding to the first network request comprises the following steps: and encrypting the first network behavior background identifier corresponding to the first network request based on a symmetric key cryptosystem. The first network behavior context identification comprises: time stamp information. The first network request is a network access request for a merchant listing page and the second network request is a network access request for a merchant details page.

After a computer instruction stored on the computer-readable storage medium provided in this embodiment is executed, after a target encryption parameter carried by a second network request is obtained, the target encryption parameter is decrypted, a first network behavior background identifier (one or more of user context information such as a user ID, a login session ID, a network device ID, and a network device IP address corresponding to the first network request, and application context information such as application channel information and application interface information) is obtained, a second network behavior background identifier corresponding to the second network request (one or more of user context information such as a user ID, a login session ID, a network device ID, and a network device IP address corresponding to the second network request, and application context information such as application channel information and application interface information) is obtained, the first network behavior background identifier is compared with the second network behavior background identifier, and whether an access subject corresponding to the second network request is a network data obtaining subject is determined based on a comparison result, the second network request is implemented based on network response data corresponding to the first network request, and the target encryption parameter is obtained after the first network request is processed for the network behavior background identifier. Through the process, the accurate identification can be carried out on the network data acquisition task scheduling system and the data acquisition operation under the acquisition strategy distribution mechanism.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

1. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.

2. As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the appended claims.

Claims

1. A network data acquisition and identification method is characterized by comprising the following steps:

acquiring a target encryption parameter carried by a second network request, wherein the second network request is realized based on network response data corresponding to a first network request, and the target encryption parameter is obtained after encryption processing is performed on a first network behavior background identifier corresponding to the first network request;

decrypting the target encryption parameter to obtain the first network behavior background identifier;

acquiring a second network behavior background identifier corresponding to the second network request;

comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject or not based on a comparison result;

wherein the network response data contains initial access policy data for target network data;

the method further comprises the following steps: obtaining target access policy data aiming at the target network data according to the initial access policy data and the target encryption parameter, and providing the target access policy data to an access subject corresponding to the first network request; or, the initial access policy data and the target encryption parameter are provided to an access subject corresponding to the first network request, so that the access subject obtains target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

2. The method of claim 1, wherein the first network behavior context identification comprises: the first network requests corresponding user context information;

the second network behavior context identification comprises: the second network requests corresponding user context information;

the comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result, includes:

comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request;

and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

3. The method of claim 2, wherein the user context information comprises at least one of: user identification information; network operating environment information;

the comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request includes:

comparing the user identification information corresponding to the first network request with the user identification information corresponding to the second network request; and/or comparing the network operation environment information corresponding to the first network request with the network operation environment information corresponding to the second network request.

4. The method of claim 3, wherein the user identification information comprises: user identity identification information, or login session identification information for anonymous users.

5. The method of claim 3, wherein the network operating environment information comprises at least one of: network device identification information; network device IP address.

6. The method of claim 1, wherein the first network behavior context identification comprises: the first network requests corresponding application context information;

the second network behavior context identification comprises: the second network requests corresponding application context information;

the comparing the first network behavior background identifier with the second network behavior background identifier, and determining whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result includes:

comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request;

and if the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

7. The method of claim 1, wherein the first network behavior context identification comprises: the first network request corresponds to user context information and the first network request corresponds to application context information;

the second network behavior context identification comprises: the second network request corresponds to user context information and the second network request corresponds to application context information;

comparing the user context information corresponding to the first network request with the user context information corresponding to the second network request, and comparing the application context information corresponding to the first network request with the application context information corresponding to the second network request;

and if the user context information corresponding to the first network request is not matched with the user context information corresponding to the second network request, and/or the application context information corresponding to the first network request is not matched with the application context information corresponding to the second network request, determining that the access subject corresponding to the second network request is a network data acquisition subject.

8. The method according to claim 6 or 7, wherein the application context information comprises at least one of: applying channel information; application interface information.

9. The method of any one of claims 2, 6, and 7, further comprising: intercepting the second network request.

10. The method of claim 1, wherein the obtaining the target encryption parameter carried by the second network request comprises: acquiring a target encryption parameter carried by a second network request aiming at the target network data;

the second network request is realized based on the network response data corresponding to the first network request, and the method comprises the following steps: the second network request is implemented based on the access policy data for the target network data contained in the network response data corresponding to the first network request.

11. The method of claim 10, further comprising:

obtaining network response data in response to the first network request;

acquiring a first network behavior background identifier corresponding to the first network request;

and encrypting the first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter.

12. The method of claim 10, wherein the access policy data for the target network data comprises: and the URL corresponding to the target network data.

13. The method of claim 12, wherein the target encryption parameter is included in a parameter of the URL.

14. The method according to claim 11, wherein the encrypting the first network behavior context identifier corresponding to the first network request comprises: and encrypting the identification information of the target network data and the first network behavior background identification.

15. The method according to claim 14, wherein the initial access policy data for the target network data is an initial URL corresponding to the target network data, and the target access policy data for the target network data is a target URL obtained after replacing identification information of the target network data included in the initial URL with the target encryption parameter;

the providing the target access policy data to the access subject corresponding to the first network request includes: and providing the target URL to an access subject corresponding to the first network request.

16. The method of claim 1, wherein providing the initial access policy data and the target encryption parameter to the access agent corresponding to the first network request comprises: and sending a data structure file containing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request.

17. The method of claim 1, wherein the target encryption parameter is included in a request header or a request body of the second network request.

18. The method of claim 10, wherein the first network request comprises: an access main body corresponding to a first network request shares the target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: the target sharing subject corresponds to a network behavior background identifier;

the encrypting the first network behavior background identifier corresponding to the first network request includes: and encrypting the user identification information of the access subject corresponding to the first network request and the network behavior background identification corresponding to the target sharing subject.

19. The method according to claim 1 or 11, wherein the encrypting the first network behavior context identifier corresponding to the first network request comprises: encrypting the first network behavior background identifier by using a target encryption key in a plurality of sets of keys, wherein the second network request comprises a target key identifier corresponding to the target encryption key;

correspondingly, the decrypting the target encryption parameter includes: and decrypting the target encryption parameter by using a target decryption key corresponding to the target key identifier.

20. The method according to claim 1 or 11, wherein the encrypting the first network behavior context identifier corresponding to the first network request comprises: and encrypting the first network behavior background identification corresponding to the first network request based on a symmetric key cryptosystem.

21. The method of claim 1, wherein the first network behavior context identification comprises: time stamp information.

22. The method of claim 1, wherein the first network request is a network access request for a merchant listing page and the second network request is a network access request for a merchant detail page.

23. A network data acquisition identification device, comprising:

a target encryption parameter obtaining unit, configured to obtain a target encryption parameter carried by a second network request, where the second network request is implemented based on network response data corresponding to a first network request, and the target encryption parameter is obtained by encrypting a first network behavior background identifier corresponding to the first network request;

the target encryption parameter decryption unit is used for decrypting the target encryption parameter to obtain the first network behavior background identifier;

a second network behavior background identifier obtaining unit, configured to obtain a second network behavior background identifier corresponding to the second network request;

a network behavior background identifier comparison unit, configured to compare the first network behavior background identifier with the second network behavior background identifier, and determine whether an access subject corresponding to the second network request is a network data acquisition subject based on a comparison result;

wherein the network response data comprises initial access policy data for target network data;

a data providing unit, configured to obtain target access policy data for the target network data according to the initial access policy data and the target encryption parameter, and provide the target access policy data to an access subject corresponding to the first network request; or, the initial access policy data and the target encryption parameter are provided to an access subject corresponding to the first network request, so that the access subject obtains target access policy data for the target network data based on the initial access policy data and the target encryption parameter.

24. The apparatus of claim 23, wherein the first network behavior context identification comprises: the first network requests corresponding user context information;

25. The apparatus of claim 24, wherein the user context information comprises at least one of: user identification information; network operating environment information;

26. The apparatus of claim 25, wherein the user identification information comprises: user identification information, or login session identification information for anonymous users.

27. The apparatus of claim 25, wherein the network operating environment information comprises at least one of: network device identification information; network device IP address.

28. The apparatus of claim 23, wherein the first network behavior context identification comprises: the first network requests corresponding application context information;

29. The apparatus of claim 23, wherein the first network behavior context identifier comprises: the first network request corresponds to user context information and the first network request corresponds to application context information;

the second network behavior context identification comprises: the second network requests the corresponding user context information and the second network requests the corresponding application context information;

30. The apparatus according to claim 28 or 29, wherein the application context information comprises at least one of: applying channel information; application interface information.

31. The apparatus of any one of claims 24, 28, 29, further comprising: and the second network request intercepting unit is used for intercepting the second network request.

32. The apparatus of claim 23, wherein the obtaining the target encryption parameter carried by the second network request comprises: acquiring a target encryption parameter carried by a second network request aiming at the target network data;

the second network request is realized based on the network response data corresponding to the first network request, and the method comprises the following steps: the second network request is realized based on the access strategy data aiming at the target network data contained in the network response data corresponding to the first network request.

33. The apparatus of claim 32, further comprising:

a network response data acquisition unit for acquiring network response data in response to the first network request;

a first network behavior background identifier obtaining unit, configured to obtain a first network behavior background identifier corresponding to the first network request;

and the encryption processing unit is used for encrypting the first network behavior background identifier corresponding to the first network request to obtain a target encryption parameter.

34. The apparatus of claim 32, wherein the access policy data for the target network data comprises: and the URL corresponding to the target network data.

35. The apparatus of claim 34, wherein the target encryption parameter is included in a parameter of the URL.

36. The apparatus of claim 33, wherein the encrypting the first network behavior context identifier corresponding to the first network request comprises: and encrypting the identification information of the target network data and the first network behavior background identification.

37. The apparatus according to claim 36, wherein the initial access policy data for the target network data is an initial URL corresponding to the target network data, and the target access policy data for the target network data is a target URL obtained after replacing identification information of the target network data included in the initial URL with the target encryption parameter;

38. The apparatus of claim 23, wherein the providing the initial access policy data and the target encryption parameter to the access agent corresponding to the first network request comprises: and sending the data structure file containing the initial access policy data and the target encryption parameter to an access subject corresponding to the first network request.

39. The apparatus of claim 23, wherein the target encryption parameter is included in a request header or a request body of the second network request.

40. The apparatus of claim 32, wherein the first network request comprises: an access main body corresponding to a first network request shares the target network data with a sharing request of a target sharing main body; the first network behavior context identifier corresponding to the first network request comprises: a network behavior background identifier corresponding to the target sharing subject;

41. The apparatus according to claim 23 or 33, wherein the encrypting the first network behavior context identifier corresponding to the first network request comprises: encrypting the first network behavior background identifier by using a target encryption key in a plurality of sets of keys, wherein the second network request comprises a target key identifier corresponding to the target encryption key;

42. The apparatus according to claim 23 or 33, wherein the encrypting the first network behavior context identifier corresponding to the first network request comprises: and encrypting the first network behavior background identifier corresponding to the first network request based on a symmetric key cryptosystem.

43. The apparatus of claim 23, wherein the first network behavior context identification comprises: time stamp information.

44. The apparatus of claim 23, wherein the first network request is a network access request for a merchant listing page, and wherein the second network request is a network access request for a merchant details page.

45. An electronic device comprising a processor and a memory; wherein,

the memory is to store one or more computer instructions, wherein the one or more computer instructions are to be executed by the processor to implement the method of any one of claims 1-22.

46. A computer-readable storage medium having stored thereon one or more computer instructions for execution by a processor to perform the method of any one of claims 1-22.