CN114491373A - Resource writing method, apparatus, electronic device and computer readable medium - Google Patents

Abstract

The embodiment of the disclosure discloses a resource writing method, a resource writing device, an electronic device and a computer readable medium. One embodiment of the method comprises: in response to determining that the target rendering page engine is started, acquiring page resources and network request resources associated with the target domain name; performing static resource filtering on the page resource to obtain a filtered resource; performing resource classification on the filtered resources and the network request resources to obtain first resources and second resources, wherein the first resources belong to the resources under the network station corresponding to the target domain name, and the second resources do not belong to the resources under the network station; performing anomaly detection on the second resource to obtain an abnormal resource and a normal resource; writing the processed normal resource and the processed first resource into a first resource queue to be uploaded, and writing the abnormal resource into a second resource queue to be uploaded. The implementation method can rapidly and efficiently write the processed resources into the corresponding queues respectively.

Description

Resource writing method, apparatus, electronic device and computer readable medium

technical field

Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a resource writing method, apparatus, electronic device, and computer-readable medium.

Background technique

Network resource collection refers to collecting various target resources (eg, link address resources, network request resources, etc.) under the target network site. Here, the abnormal target resources are screened out from the collected target resources, which can be subsequently used to ensure the network security of the web (World Wide Web, World Wide Web) system. For the collection of target resources, a method is usually adopted: relevant developers actively trigger the collection of various target resources related to the target network site.

However, when using the above method to collect various target resources, there are often the following technical problems:

Manual collection has the problem that the workload is large and the efficiency of collecting target resources is too low. In addition, the low efficiency of target resource collection will greatly affect the security of the web system.

SUMMARY OF THE INVENTION

This summary of the disclosure serves to introduce concepts in a simplified form that are described in detail in the detailed description that follows. The content section of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

Some embodiments of the present disclosure propose a resource writing method, apparatus, electronic device, and computer-readable medium to solve the technical problems mentioned in the background section above.

In a first aspect, some embodiments of the present disclosure provide a resource writing method, comprising: in response to determining that the target rendering page engine is started, obtaining a target domain name associated with the engine according to the engine resources loaded by the target rendering page engine during the startup process performing static resource filtering on the above-mentioned page resources to obtain filtered resources; performing resource classification on the above-mentioned filtered resources and the above-mentioned network request resources to obtain the first resource and the second resource, wherein the above-mentioned first resource The resource is a resource belonging to the network site corresponding to the target domain name, and the second resource is a resource that does not belong to the network site; abnormality detection is performed on the second resource to obtain abnormal resources and normal resources; and the processed first resource is written into the first resource queue to be uploaded, and the above-mentioned abnormal resource is written into the second resource queue to be uploaded.

Optionally, the above method further includes: uploading the resources in the first resource queue and the resources in the second resource queue to the target server.

Optionally, the above-mentioned engine resources include: the engine resources of the interface-less mode engine; and the above-mentioned engine resources loaded according to the above-mentioned target rendering page engine during the startup process, and obtaining the page resources and network request resources associated with the target domain name, including: obtaining The tab page associated with the target domain name; using the interfaceless mode engine to obtain the page resource associated with the tab page; according to the page resource associated with the tab page, obtain the page resource associated with the target domain name and the network request resource. .

Optionally, before writing the processed normal resources and the processed first resources into the first resource queue to be uploaded, and before writing the above-mentioned abnormal resources into the second resource queue to be uploaded, the method further includes: Resource deduplication processing is performed on the normal resource and the first resource to obtain the normal resource after processing and the first resource after processing.

Optionally, obtaining the page resource associated with the target domain name and the network request resource according to the page resource associated with the label page includes: in response to determining that there is a link resource in the page resource associated with the label page, determining the above-mentioned page resource. at least one link resource in the page resources associated with the tab page; intercept the network request corresponding to the tab page to obtain the network request resource corresponding to the tab page; for each link resource in the at least one link resource, execute the page resource Obtaining steps: generating a tab page corresponding to the above-mentioned link resource as a target tab page; obtaining page resources associated with the above-mentioned target tab page; intercepting the network request corresponding to the above-mentioned target tab page, so as to obtain the network corresponding to the above-mentioned target tab page requesting resources; determining whether there is a link resource in the page resource associated with the above-mentioned target tab page; in response to determining that there is no link resource in the page resource associated with the above-mentioned target tab page, link the page resource associated with the above-mentioned target tab page with the above-mentioned target tab page The associated page resource is determined as a sub-page resource, and the network request resource corresponding to the above-mentioned tab page and the network request resource corresponding to the above-mentioned target tab page are combined to obtain a combined resource as a sub-network request resource, wherein the above-mentioned sub-page resource is a resource in the page resource associated with the target domain name, and the sub-network request resource is a resource in the network request resource associated with the target domain name.

Optionally, the above method further includes: in response to determining that there is a link resource in the page resources associated with the target tab page, determining at least one link resource in the page resources associated with the target tab page, and continuing to perform the page resource acquisition. step.

Optionally, the above method further includes: in response to determining that there is no link resource in the page resource associated with the above-mentioned tab page, determining the page resource associated with the above-mentioned tab page as the page resource associated with the above-mentioned target domain name, and adding the above-mentioned label page resource. The network request resource corresponding to the page is determined to be the network request resource associated with the target domain name.

In a second aspect, some embodiments of the present disclosure provide a resource writing apparatus, including: an acquisition unit configured to, in response to determining that a target rendering page engine is started, render engine resources loaded by the page engine during the startup process according to the above target rendering , obtain the page resources and network request resources associated with the target domain name; the resource filtering unit is configured to perform static resource filtering on the above page resources to obtain filtered resources; the resource classification unit is configured to filter the above-mentioned resources and the above-mentioned network resources. Requesting resources to perform resource classification to obtain a first resource and a second resource, wherein the first resource is a resource belonging to a network site corresponding to the target domain name, and the second resource is a resource that does not belong to the network site; anomaly detection unit , is configured to perform anomaly detection on the above-mentioned second resources to obtain abnormal resources and normal resources; the resource writing unit is configured to write the processed normal resources and the processed first resources into the first resource to be uploaded queue, and write the above-mentioned abnormal resource into the second resource queue to be uploaded.

Optionally, the above-mentioned apparatus further includes: uploading the resources in the above-mentioned first resource queue and the resources in the above-mentioned second resource queue to the target server.

Optionally, the above-mentioned engine resources include: the engine resources of the interfaceless mode engine; and the acquiring unit is configured to: acquire the tab page associated with the above-mentioned target domain name; use the above-mentioned interfaceless mode engine to obtain the page resource associated with the above-mentioned tabbed page ; Obtain the page resources associated with the target domain name and the network request resources according to the page resources associated with the label page.

Optionally, the above-mentioned apparatus further includes: performing resource deduplication processing on the above-mentioned normal resources and the above-mentioned first resources, so as to obtain the processed normal resources and the processed first resources.

Optionally, the obtaining unit is configured to: in response to determining that there is a link resource in the page resources associated with the above-mentioned tab page, determine at least one link resource in the page resources associated with the above-mentioned tab page; intercept the network request corresponding to the above-mentioned tab page. , in order to obtain the network request resource corresponding to the above-mentioned tag page; for each link resource in the above-mentioned at least one link resource, perform the page resource acquisition step: generate a tag page corresponding to the above-mentioned link resource as the target tag page; The page resource associated with the target tab page; intercepting the network request corresponding to the target tab page to obtain the network request resource corresponding to the target tab page; determining whether the page resource associated with the target tab page has a link resource; in response to determining the above There is no link resource in the page resource associated with the target tab page, the page resource associated with the above tab page and the page resource associated with the target tab page are determined as sub-page resources, and the network request resource corresponding to the tab page and the page resource are determined. The network request resources corresponding to the above-mentioned target tag pages are combined to obtain the combined resources as sub-network request resources, wherein the above-mentioned sub-page resources are resources in the page resources associated with the above-mentioned target domain name, and the above-mentioned sub-network request resources are the above-mentioned target domain name. The resource in the associated network request resource.

Optionally, the obtaining unit is configured to: in response to determining that there is a link resource in the page resource associated with the target tab page, determine at least one link resource in the page resource associated with the target tab page, and continue to execute the page resource. Get steps.

Optionally, the obtaining unit is configured to: in response to determining that there is no link resource in the page resource associated with the above-mentioned tab page, determine the page resource associated with the above-mentioned tab page as the page resource associated with the above-mentioned target domain name, and The network request resource corresponding to the label page is determined as the network request resource associated with the target domain name.

In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, when one or more programs are stored by one or more The processor executes such that the one or more processors implement a method as described in any implementation of the first aspect.

In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any implementation manner of the first aspect.

The foregoing embodiments of the present disclosure have the following beneficial effects: the resource writing methods of some embodiments of the present disclosure can quickly and efficiently write processed resources into corresponding queues respectively. Specifically, the reason for the inefficient writing of resources is that manual collection has a large workload and the efficiency of collecting target resources is too low. As a result, resource writing is not efficient enough. Based on this, the resource writing methods of some embodiments of the present disclosure can respond to determining that the target rendering page engine is started, and can efficiently obtain page resources associated with the target domain name according to the engine resources loaded by the target rendering page engine during the startup process. and network request resources. Here, the above-mentioned page resources and network request resources include resources to be uploaded. Then, static resource filtering is performed on the above page resources to obtain filtered resources. Here, the static resources in the page resources have little effect on ensuring the network security of the web system. Therefore, removing the static resources with less effect from the page resources can more efficiently ensure the network security of the web system. Further, the above-mentioned filtered resources and the above-mentioned network request resources are classified as resources to obtain a first resource and a second resource, wherein the above-mentioned first resource is a resource belonging to the network site corresponding to the above-mentioned target domain name, and the above-mentioned second resource is not belonging to Resources under the aforementioned web site. Here, abnormal resources can be efficiently distinguished by resource classification. Next, abnormality detection is performed on the above-mentioned second resource, so that abnormal resources and normal resources can be obtained efficiently. Finally, the processed normal resources and the processed first resources are written into the first resource queue to be uploaded, and the above abnormal resources are written into the second resource queue to be uploaded. Through the first resource queue and the second resource queue, the network security of the web system can be effectively guaranteed subsequently.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

1 is a schematic diagram of an application scenario of a resource writing method according to some embodiments of the present disclosure;

2 is a flowchart of some embodiments of resource writing methods according to the present disclosure;

3 is a flowchart of other embodiments of resource writing methods according to the present disclosure;

4 is a schematic diagram of a tab page in some embodiments of the resource writing method according to the present disclosure;

5 is a schematic structural diagram of some embodiments of a resource writing apparatus according to the present disclosure;

6 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings. The embodiments of this disclosure and features of the embodiments may be combined with each other without conflict.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

FIG. 1 is a schematic diagram of an application scenario of a resource writing method according to some embodiments of the present disclosure.

In the application scenario of FIG. 1 , in response to determining that the target rendering page engine 102 is started, the client 101 may first obtain the page resource 105 associated with the target domain name 104 according to the engine resource 103 loaded by the target rendering page engine 102 during the startup process. and network request resource 106. Then, the client 101 may perform static resource filtering on the above-mentioned page resource 105 to obtain the filtered resource 107 . Next, the client 101 may perform resource classification on the above-mentioned filtered resource 107 and the above-mentioned network request resource 106 to obtain the first resource 108 and the second resource 109 . The above-mentioned first resource 108 is a resource belonging to the network site corresponding to the above-mentioned target domain name 104, and the above-mentioned second resource 109 is a resource that does not belong to the above-mentioned network site. Further, the client 101 may perform anomaly detection on the second resource 109 to obtain an anomaly resource 110 and a normal resource 111 . Finally, the client 101 can write the processed normal resource 113 and the processed first resource 112 into the first resource queue 114 to be uploaded, and write the above-mentioned abnormal resource 110 into the second resource queue 115 to be uploaded.

It should be noted that the client can be hardware or software. When the client is hardware, it can be implemented as a distributed device cluster composed of multiple devices, or as a single server/single device. When the client is software, it may be implemented as multiple software or software modules for providing distributed services, or may be implemented as a single software or software module. There is no specific limitation here.

It should be understood that the number of clients in FIG. 1 is merely illustrative. There can be any number of clients depending on the implementation needs.

With continued reference to FIG. 2, a flow 200 of some embodiments of resource writing methods according to the present disclosure is shown. The resource writing method includes the following steps:

Step 201 , in response to determining that the target rendering page engine is started, acquire page resources and network request resources associated with the target domain name according to the engine resources loaded by the target rendering page engine during the startup process.

In some embodiments, in response to determining that the target rendering page engine is started, the execution body of the above resource writing method (for example, the client 101 shown in FIG. 1 ) may render engine resources loaded by the page engine during the startup process according to the above target rendering method, Get the page resources and network request resources associated with the target domain name. Wherein, the above-mentioned target rendering page engine may be a Go-Driver engine. The above-mentioned Go-Driver engine may be a server-side rendering page engine developed by using the go language and the target browser engine. Here, the above-mentioned Go-Driver engine can render the incoming URL (Uniform ResourceLocator, Uniform Resource Locator) to simulate the scene of opening the page, so that there is no need to manually open the browser to obtain the corresponding resources. The above target domain name may be a domain name corresponding to the target website. The above-mentioned page resources may be various types of resources of the page corresponding to the target domain name. The above-mentioned page resources may include, but are not limited to, at least one of the following: static resources of the page corresponding to the target domain name, and link resources of the page corresponding to the target domain name. The above-mentioned network request resource may be a data request interface (Application Programming Interface, API) resource. The existence of the above target domain name corresponds to the target Internet Protocol (IP, Internet Protocol). The corresponding page can be obtained through the target Internet Internet Protocol. The above engine resources are the resources to be configured by the Go-Driver engine during startup. The above engine resources may include, but are not limited to, at least one of the following: resources corresponding to resource collectors, resources corresponding to resource deduplication filters. The configuration of engine resources here may also include: configuration of central processing unit (Central Processing Unit, CPU) resources, and so on.

Step 202: Perform static resource filtering on the above page resources to obtain filtered resources.

In some embodiments, the above-mentioned execution body may perform static resource filtering on the above-mentioned page resources to obtain filtered resources. The above-mentioned static resources may include, but are not limited to, at least one of the following: picture resources, video resources.

As an example, the above-mentioned execution body may remove the static resources in the page resources to obtain the filtered resources.

Here, the purpose of removing static resources is that static resources have little effect on the determination of web system security, so removing static resources can greatly improve the efficiency of ensuring web system security.

Step 203: Perform resource classification on the above-mentioned filtered resources and the above-mentioned network request resources to obtain a first resource and a second resource.

In some embodiments, the above-mentioned execution body may perform resource classification on the above-mentioned filtered resources and the above-mentioned network request resources to obtain the first resource and the second resource. The first resource is a resource that belongs to the network site corresponding to the target domain name, and the second resource is a resource that does not belong to the network site.

As an example, the above-mentioned executive body may use a preset resource division rule to classify the above-mentioned filtered resources and the above-mentioned network request resources, and obtain the first resource and the second resource.

Step 204: Perform abnormality detection on the above-mentioned second resource to obtain abnormal resources and normal resources.

In some embodiments, the above-mentioned execution body may perform abnormality detection on the above-mentioned second resource, so as to obtain abnormal resources and normal resources.

As an example, the above-mentioned execution body may first obtain the website corresponding to the target domain name. Then, the above-mentioned executive body may determine the resource list corresponding to the above-mentioned web site. Finally, the abnormal resource and the normal resource are obtained by comparing the above-mentioned second resource and the above-mentioned resource list. Wherein, the sub-resources in the above-mentioned second resource that do not exist in the above-mentioned resource list are abnormal resources.

Step 205: Write the processed normal resource and the processed first resource into the first resource queue to be uploaded, and write the above abnormal resource into the second resource queue to be uploaded.

In some embodiments, the above-mentioned execution body may write the processed normal resources and the processed first resources into the first resource queue to be uploaded, and write the above-mentioned abnormal resources into the second resource queue to be uploaded. The above-mentioned processed normal resources may be resources processed on normal resources. The above-mentioned processed first resource may be a resource processed on the first resource.

As an example, the above-mentioned execution body may perform resource supplementation on the normal resource and the first resource, so as to obtain the processed normal resource and the processed first resource.

In some optional implementation manners of some embodiments, the execution body may upload the resources in the first resource queue and the resources in the second resource queue to the target server. The above-mentioned target server can manage each client in a unified manner. Each of the above clients may be deployed in a cluster.

In some optional implementations of some embodiments, the above-mentioned normal resources after processing and the processed first resources are written into the first resource queue to be uploaded, and the above-mentioned abnormal resources are written into the second to-be-uploaded resource queue. Before the resource queue, the above steps also include:

The execution body performs resource deduplication processing on the normal resource and the first resource, and obtains the processed normal resource and the processed first resource.

The foregoing embodiments of the present disclosure have the following beneficial effects: the resource writing methods of some embodiments of the present disclosure can quickly and efficiently write processed resources into corresponding queues respectively. Specifically, the reason for the inefficient writing of resources is that manual collection has a large workload and the efficiency of collecting target resources is too low. As a result, resource writing is not efficient enough. Based on this, the resource writing methods of some embodiments of the present disclosure can respond to determining that the target rendering page engine is started, and can efficiently obtain page resources associated with the target domain name according to the engine resources loaded by the target rendering page engine during the startup process. and network request resources. Here, the above-mentioned page resources and network request resources include resources to be uploaded. Then, static resource filtering is performed on the above page resources to obtain filtered resources. Here, the static resources in the page resources have little effect on ensuring the network security of the web system. Therefore, removing the static resources with less effect from the page resources can more efficiently ensure the network security of the web system. Further, the above-mentioned filtered resources and the above-mentioned network request resources are classified as resources to obtain a first resource and a second resource, wherein the above-mentioned first resource is a resource belonging to the network site corresponding to the above-mentioned target domain name, and the above-mentioned second resource is not belonging to Resources under the aforementioned web site. Here, abnormal resources can be efficiently distinguished by resource classification. Next, abnormality detection is performed on the above-mentioned second resource, so that abnormal resources and normal resources can be obtained efficiently. Finally, the processed normal resources and the processed first resources are written into the first resource queue to be uploaded, and the above abnormal resources are written into the second resource queue to be uploaded. Through the first resource queue and the second resource queue, the network security of the web system can be effectively guaranteed subsequently.

With further reference to FIG. 3, a flow 300 of other embodiments of resource writing methods according to the present disclosure is shown. The resource writing method includes the following steps:

Step 301: Acquire a tab page associated with the target domain name.

In some embodiments, the execution body (eg, the client 101 shown in FIG. 1 ) can obtain the tag page associated with the target domain name. The engine resources include: engine resources of the interfaceless mode engine. The above interfaceless mode engine may be a Headless engine.

As an example, the above-mentioned execution body may obtain the tab page associated with the above-mentioned target domain name through the following steps:

The first step is to perform domain name resolution on the target domain name to obtain the IP address corresponding to the target domain name.

The second step is to obtain the label page corresponding to the target domain name through the pre-packaged label page management engine according to the IP address.

As an example, as shown in FIG. 4, FIG. 4 shows a schematic diagram of a tab page.

Step 302, using the above-mentioned no-interface mode engine to acquire the page resource associated with the above-mentioned tab page.

In some embodiments, the above-mentioned execution body may use the above-mentioned interfaceless mode engine to obtain the page resource associated with the above-mentioned tab page.

As an example, the above-mentioned execution body may obtain the page resource associated with the above-mentioned tab page by using the automated page click script pre-packaged in the above-mentioned interfaceless mode engine. The above-mentioned automated page click script may be a JS script. Among them, when the no-interface mode engine is started, the domain name, Cookie (data stored on the user's local terminal), Host (host) information, and JS automatic click script are encapsulated into the no-interface mode engine together.

Step 303: Acquire the page resource associated with the target domain name and the network request resource according to the page resource associated with the tag page.

In some embodiments, the execution body may acquire the page resource associated with the target domain name and the network request resource according to the page resource associated with the tag page.

As an example, the above-mentioned execution body may first intercept the network request corresponding to the tab page. Then, the execution body may determine the network request corresponding to the tab page as the network request resource corresponding to the target domain name, and determine the page resource associated with the tab page as the page resource associated with the target domain name.

In some optional implementations of some embodiments, obtaining the page resource associated with the target domain name and the network request resource according to the page resource associated with the label page may include the following steps:

In the first step, in response to determining that there is a link resource in the page resource associated with the tab page, the execution body may determine at least one link resource in the page resource associated with the tab page.

As an example, the above-mentioned execution body may determine at least one link resource in the page resources associated with the above-mentioned tab page by means of page query.

In the second step, the above-mentioned execution body intercepts the network request corresponding to the above-mentioned tab page, so as to obtain the network request resource corresponding to the above-mentioned tab page.

The third step, for each link resource in the above at least one link resource, execute the page resource acquisition step:

In the first sub-step, a tab page corresponding to the above linked resource is generated as a target tab page.

The second sub-step is to acquire page resources associated with the above target tab page.

As an example, the above-mentioned execution body may use the above-mentioned automated page click script to acquire the page resource associated with the above-mentioned target tab page.

In the third sub-step, the execution body may intercept the network request corresponding to the target tab page, so as to obtain the network request resource corresponding to the target tab page.

In the fourth sub-step, the execution body may determine whether there is a link resource in the page resource associated with the target tab page.

As an example, the above-mentioned execution body may determine whether there is a link resource in the page resource associated with the above-mentioned target tab page by means of a link query.

In the fifth sub-step, in response to determining that there is no link resource in the page resource associated with the target tab page, the execution body may determine the page resource associated with the target tab page and the page resource associated with the target tab page as a sub-page resource, and combining the network request resource corresponding to the above tab page and the network request resource corresponding to the target tab page to obtain the combined resource as the sub-network request resource. The above-mentioned sub-page resources are resources in the page resources associated with the above-mentioned target domain name. The above-mentioned sub-network request resource is a resource in the network request resource associated with the above-mentioned target domain name.

Optionally, the above steps also include:

In response to determining that there is a link resource in the page resource associated with the target tab page, the execution body determines at least one link resource in the page resource associated with the target tab page, and continues to perform the page resource obtaining step.

Optionally, the above steps also include:

In response to determining that there is no link resource in the page resource associated with the above-mentioned tab page, the above-mentioned execution body may determine the page resource associated with the above-mentioned tab page as the page resource associated with the above-mentioned target domain name, and send the network request corresponding to the above-mentioned tab page. The resource is determined to be the network request resource associated with the above target domain name.

Step 304: Perform static resource filtering on the above page resources to obtain filtered resources.

Step 305: Perform resource classification on the above-mentioned filtered resources and the above-mentioned network request resources to obtain a first resource and a second resource.

Step 306: Perform abnormality detection on the above-mentioned second resource to obtain abnormal resources and normal resources.

Step 307: Write the processed normal resource and the processed first resource into the first resource queue to be uploaded, and write the above-mentioned abnormal resource into the second resource queue to be uploaded.

In some embodiments, for the specific implementation of steps 304-307 and the technical effects brought about by them, reference may be made to steps 202-205 in the embodiment corresponding to FIG. 2, and details are not repeated here.

As can be seen from FIG. 3 , compared with the description of some embodiments corresponding to FIG. 2 , the process 300 of the resource writing method in some embodiments corresponding to FIG. 3 more highlights the acquisition of the target domain name association according to the interfaceless mode engine The specific steps of the page resources and network request resources. Therefore, the solutions described in these embodiments can efficiently and quickly realize automatic acquisition of page resources and network request resources without requiring manual clicks.

Further referring to FIG. 5 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a resource writing apparatus, these apparatus embodiments correspond to those method embodiments shown in FIG. 2 , the apparatus Specifically, it can be applied to various electronic devices.

As shown in FIG. 5 , a resource writing device 500 includes: an acquisition unit 501 , a resource filtering unit 502 , a resource classification unit 503 , an abnormality detection unit 504 and a resource writing unit 505 . The obtaining unit 501 is configured to, in response to determining that the target rendering page engine is started, obtain the page resources and network request resources associated with the target domain name according to the engine resources loaded by the target rendering page engine during the startup process; the resource filtering unit 502 , is configured to perform static resource filtering on the above-mentioned page resources to obtain the filtered resources; the resource classification unit 503 is configured to perform resource classification on the above-mentioned filtered resources and the above-mentioned network request resources to obtain the first resource and the second resource, wherein , the above-mentioned first resource is a resource that belongs to the network site corresponding to the above-mentioned target domain name, and the above-mentioned second resource is a resource that does not belong to the above-mentioned network site; the abnormality detection unit 504 is configured to perform abnormality detection on the above-mentioned second resource to obtain Abnormal resources and normal resources; the resource writing unit 505 is configured to write the processed normal resources and the processed first resources into the first resource queue to be uploaded, and write the above abnormal resources into the second to-be-uploaded resource queue resource queue.

In some optional implementations of some embodiments, the above-mentioned apparatus 500 further includes: an uploading unit (not shown in the figure). The uploading unit may be configured to: upload the resources in the first resource queue and the resources in the second resource queue to the target server.

In some optional implementations of some embodiments, the above-mentioned engine resources include: engine resources of the interfaceless mode engine, and the acquiring unit 501 is further configured to: acquire the tab page associated with the above-mentioned target domain name; use the above-mentioned interfaceless mode The schema engine obtains the page resources associated with the above-mentioned tab pages; according to the page resources associated with the above-mentioned tab pages, obtains the page resources associated with the above-mentioned target domain name and the above-mentioned network request resources.

In some optional implementations of some embodiments, the above-mentioned apparatus 500 further includes: a deduplication unit (not shown in the figure). The deduplication unit may be configured to: perform resource deduplication processing on the normal resources and the first resources to obtain the processed normal resources and the processed first resources.

In some optional implementations of some embodiments, the obtaining unit 501 is further configured to: in response to determining that a link resource exists in the page resource associated with the above-mentioned tab page, determine at least one of the page resources associated with the above-mentioned tab page a link resource; intercept the network request corresponding to the above-mentioned tab page to obtain the network request resource corresponding to the above-mentioned tab page; for each link resource in the above-mentioned at least one link resource, perform a page resource acquisition step: generate a page resource corresponding to the above-mentioned link resource The tab page is used as the target tab page; obtain the page resources associated with the above-mentioned target tab page; intercept the network request corresponding to the above-mentioned target tab page to obtain the network request resource corresponding to the above-mentioned target tab page; determine that the above-mentioned target tab page is associated with Whether there is a link resource in the page resource of the above-mentioned target tab page; in response to determining that there is no link resource in the page resource associated with the above-mentioned target tab page, determine the page resource associated with the above-mentioned tab page and the page resource associated with the above-mentioned target tab page as a sub-page resource , and combine the network request resource corresponding to the above-mentioned tab page and the network request resource corresponding to the above-mentioned target tab page to obtain the combined resource as a sub-network request resource, wherein the above-mentioned sub-page resource is one of the page resources associated with the above-mentioned target domain name The above-mentioned sub-network request resource is a resource in the network request resource associated with the above-mentioned target domain name.

In some optional implementations of some embodiments, the obtaining unit 501 is further configured to: in response to determining that a link resource exists in the page resource associated with the target tab page, determine whether the page resource associated with the target tab page is in the page resource. at least one link resource, and continue to perform the above steps of obtaining page resources.

In some optional implementations of some embodiments, the obtaining unit 501 is further configured to: in response to determining that there is no link resource in the page resource associated with the above tab page, determine the page resource associated with the above tab page as The page resource associated with the target domain name, and the network request resource corresponding to the label page is determined as the network request resource associated with the target domain name.

It can be understood that the units recorded in the apparatus 500 correspond to the respective steps in the method described with reference to FIG. 2 . Therefore, the operations, features and beneficial effects described above with respect to the method are also applicable to the apparatus 500 and the units included therein, and details are not described herein again.

Referring next to FIG. 6 , it shows a schematic structural diagram of an electronic device (eg, client 101 in FIG. 1 ) 600 suitable for implementing some embodiments of the present disclosure. The electronic device shown in FIG. 6 is just an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 6 , an electronic device 600 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 601 that may be loaded into random access according to a program stored in a read only memory (ROM) 602 or from a storage device 608 Various appropriate actions and processes are executed by the programs in the memory (RAM) 603 . In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are also stored. The processing device 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

Typically, the following devices can be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 607 of a computer, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. Communication means 609 may allow electronic device 600 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 6 shows electronic device 600 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 6 may represent one device, or may represent multiple devices as required.

In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In some such embodiments, the computer program may be downloaded and installed from the network via the communication device 609 , or from the storage device 608 , or from the ROM 602 . When the computer program is executed by the processing device 601, the above-described functions defined in the methods of some embodiments of the present disclosure are performed.

It should be noted that, in some embodiments of the present disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the foregoing two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Rather, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the client and server can use any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) to communicate, and can communicate with digital data in any form or medium Communication (eg, a communication network) interconnects. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: in response to determining that the target rendering page engine is started, according to the above-mentioned target rendering page engine in the startup process The engine resources loaded in the target domain name are obtained, and the page resources and network request resources associated with the target domain name are obtained; the static resource filtering is performed on the above page resources to obtain the filtered resources; resources and a second resource, wherein the first resource is a resource belonging to a network site corresponding to the target domain name, and the second resource is a resource that does not belong to the network site; anomaly detection is performed on the second resource to obtain anomalies resources and normal resources; write the processed normal resources and the processed first resources into the first resource queue to be uploaded, and write the above abnormal resources into the second resource queue to be uploaded.

Computer program code for carrying out operations of some embodiments of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, or a combination thereof, Also included are conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider to via Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by means of software, and may also be implemented by means of hardware. The described unit can also be set in the processor, for example, it can be described as: a processor includes an acquisition unit, a resource filtering unit, a resource classification unit, an anomaly detection unit and a resource writing unit. Wherein, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the acquisition unit may also be described as "in response to determining that the target rendering page engine is started, according to the above target rendering page engine during the startup process Loaded engine resources, obtain the page resources associated with the target domain name and the unit of network request resources".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.

The above descriptions are merely some preferred embodiments of the present disclosure and illustrations of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned inventive concept, the above-mentioned Other technical solutions formed by any combination of technical features or their equivalent features. For example, a technical solution is formed by replacing the above-mentioned features with the technical features disclosed in the embodiments of the present disclosure (but not limited to) with similar functions.

Claims (10)

1. A resource writing method, comprising: In response to determining that the target rendering page engine is started, obtain the page resources and network request resources associated with the target domain name according to the engine resources loaded by the target rendering page engine during the startup process; Perform static resource filtering on the page resources to obtain filtered resources; Perform resource classification on the filtered resource and the network request resource to obtain a first resource and a second resource, wherein the first resource is a resource belonging to a network site corresponding to the target domain name, and the second resource For resources that do not belong to the said network site; performing anomaly detection on the second resource to obtain anomalous resources and normal resources; The processed normal resources and the processed first resources are written into the first resource queue to be uploaded, and the abnormal resources are written into the second resource queue to be uploaded. 2. The method of claim 1, wherein the method further comprises: Upload the resources in the first resource queue and the resources in the second resource queue to the target server. 3. The method of claim 1, wherein the engine resources comprise: engine resources of a no-interface mode engine; and The rendering of the engine resources loaded by the page engine during the startup process according to the target, and obtaining the page resources and network request resources associated with the target domain name, include: obtain the tag page associated with the target domain name; Utilize the interfaceless mode engine to obtain the page resource associated with the tab page; According to the page resource associated with the tab page, the page resource associated with the target domain name and the network request resource are acquired. 4. The method according to claim 1, wherein, in the process of writing the processed normal resources and the processed first resources into the first resource queue to be uploaded, and writing the abnormal resources into the to-be-uploaded first resource queue Before the second resource queue, the method further includes: Perform resource deduplication processing on the normal resource and the first resource to obtain the processed normal resource and the processed first resource. 5. The method according to claim 3, wherein the obtaining the page resource associated with the target domain name and the network request resource according to the page resource associated with the label page comprises: In response to determining that a linked resource exists in the page resource associated with the tab page, determining at least one linked resource in the page resource associated with the tab page; Intercepting the network request corresponding to the tab page to obtain the network request resource corresponding to the tab page; For each linked resource in the at least one linked resource, a page resource acquisition step is performed: generating a tab page corresponding to the linked resource as a target tab page; obtaining page resources associated with the target tab page; Intercepting the network request corresponding to the target tab page to obtain the network request resource corresponding to the target tab page; determining whether there is a link resource in the page resource associated with the target tab page; In response to determining that a linked resource does not exist in the page resource associated with the target tab page, determining the page resource associated with the tab page and the page resource associated with the target tab page as sub-page resources, and determining the page resource associated with the target tab page The network request resource corresponding to the tab page and the network request resource corresponding to the target tab page are combined to obtain a combined resource as a sub-network request resource, wherein the sub-page resource is one of the page resources associated with the target domain name resource, the sub-network request resource is a resource in the network request resource associated with the target domain name. 6. The method of claim 5, wherein the method further comprises: In response to determining that a link resource exists in the page resources associated with the target tab page, determining at least one link resource in the page resources associated with the target tab page, and continuing to perform the page resource obtaining step. 7. The method of claim 5, wherein the method further comprises: In response to determining that there is no link resource in the page resource associated with the label page, determining the page resource associated with the label page as the page resource associated with the target domain name, and sending a network request corresponding to the label page The resource is determined to be the network request resource associated with the target domain name. 8. A resource writing device, comprising: an obtaining unit, configured to, in response to determining that the target rendering page engine is started, obtain the page resources and network request resources associated with the target domain name according to the engine resources loaded by the target rendering page engine during the startup process; a resource filtering unit, configured to perform static resource filtering on the page resources to obtain filtered resources; The resource classification unit is configured to perform resource classification on the filtered resources and the network request resources to obtain a first resource and a second resource, wherein the first resource belongs to the network site corresponding to the target domain name. resource, the second resource is a resource that does not belong to the network site; an abnormality detection unit, configured to perform abnormality detection on the second resource to obtain abnormal resources and normal resources; The resource writing unit is configured to write the processed normal resources and the processed first resources into the first resource queue to be uploaded, and to write the abnormal resources into the second resource queue to be uploaded. 9. An electronic device comprising: one or more processors; a storage device on which one or more programs are stored, The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7. 10. A computer-readable medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method of any one of claims 1-7.

Images