WO2019071966A1

WO2019071966A1 - Crawler data-based user behavior analysis method, application server and readable storage medium

Info

Publication number: WO2019071966A1
Application number: PCT/CN2018/089707
Authority: WO
Inventors: 蔡俊
Original assignee: 平安科技（深圳）有限公司
Priority date: 2017-10-13
Filing date: 2018-06-03
Publication date: 2019-04-18
Also published as: CN107870986A

Abstract

Disclosed in the present application is a crawler data-based user behavior analysis method, the method comprising: pre-configuring a crawler application program; detecting whether the crawler application program is started; after the crawler application program is started, using the crawler application program to acquire crawler information; analyzing the crawler information and extracting a keyword; using the keyword to tag a user; and performing statistics on user behavior according to the tag. The present application further provides an application server and a computer readable storage medium. In the application server and crawler data-based user behavior analysis method provided by the present application, the range of user data is preliminarily delimited; in addition, tagging of the user data is processed in clustering, so as to acquire the user behavior, thereby improving the accuracy of data analysis, facilitating the business decision-making, and improving the marketing efficiency.

Description

User behavior analysis method based on crawler data, application server and readable storage medium

This application is based on the priority of the Paris Convention on the application of the Chinese Patent Application No. CN 201710951681.3 filed on October 13, 2017, entitled "User Behavior Analysis Method Based on Reptile Data, Application Server and Computer Readable Storage Media", The entire contents of the Chinese patent application are incorporated herein by reference.

Technical field

The present application relates to the field of communications technologies, and in particular, to a user behavior analysis method based on crawler data, an application server, and a readable storage medium.

Background technique

Big data enables companies to easily access a wider range of feedback from users over the Internet, providing an adequate data base for further accurate and rapid analysis of important business information such as user behavior habits and spending habits. Along with the deepening of people's understanding, a concept emerges quietly: User Profile, a perfect abstraction of a user's information, can be seen as the foundation of enterprise application big data.

However, the current big data acquisition, the amount of data is large and complex, can not achieve the initial scope of the data, and thus affect the accuracy of subsequent user behavior analysis.

Summary of the invention

The application provides a user behavior analysis method and an application server based on crawler data, which preliminarily delimits the scope of user data, and simultaneously labels the user data in a clustering process to obtain user behavior and improve the accuracy of data analysis. To help business decisions and improve marketing efficiency.

The first aspect of the present application provides a user behavior analysis method based on crawler data, which is applied to an application server, and the method includes:

Pre-configure the crawler application;

Detecting whether the crawler application is enabled;

After the crawler application is opened, the crawler application is used to obtain crawler information;

Analyzing the crawler information and extracting keywords;

Using the keywords to tag users; and

User behavior statistics are performed based on the tags.

A second aspect of the present application provides an application server, where the application server includes a memory, a processor, and a crawler data-based user behavior analysis program executable on the processor, the crawler data is stored on the memory The user behavior analysis program is implemented by the processor to implement the following steps:

Pre-configure the crawler application;

Detecting whether the crawler application is enabled;

Analyzing the crawler information and extracting keywords;

Using the keywords to tag users; and

User behavior statistics are performed based on the tags.

A third aspect of the present application provides a computer readable storage medium storing a user behavior analysis program based on crawler data, the crawler data-based user behavior analysis program being executable by at least one processor, Taking the at least one processor to perform the following steps:

Pre-configure the crawler application;

Detecting whether the crawler application is enabled;

Analyzing the crawler information and extracting keywords;

Using the keywords to tag users; and

User behavior statistics are performed based on the tags.

Compared with the prior art, the application server, the crawler data-based user behavior analysis method, and the computer readable storage medium proposed by the present application firstly configure a crawler application first; secondly, detect whether the crawler application is enabled; Then, after the crawler application is opened, again, the crawler application is used to acquire crawler information; then, the crawler information is analyzed and keywords are extracted; further, the keyword is used to tag the user; finally, User behavior statistics are performed based on the tags. In this way, the large amount of data acquired by the big data in the prior art can be avoided, and the preliminary range delineation of the data cannot be performed, thereby affecting the drawbacks of the accuracy of subsequent user behavior analysis. By preliminarily delineating the scope of user data, and simultaneously tagging user data in cluster processing, obtaining user behavior, improving the accuracy of data analysis, contributing to business decisions and improving marketing efficiency.

DRAWINGS

1 is a schematic diagram of an optional hardware architecture of an application server;

2 is a program block diagram of a first embodiment of a user behavior analysis program based on crawler data of the present application;

3 is a flowchart of a first embodiment of a user behavior analysis method based on crawler data according to the present application;

FIG. 4 is a flowchart of a second embodiment of a user behavior analysis method based on crawler data according to the present application.

Reference mark:

The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings.

Detailed ways

The principles and features of the present application are described in the following with reference to the accompanying drawings, which are only used to explain the present application and are not intended to limit the scope of the application.

Referring to FIG. 1, it is a schematic diagram of an optional hardware architecture of the application server 1.

The application server 1 may be a computing device such as a rack server, a blade server, a tower server, or a rack server. The application server 1 may be a stand-alone server or a server cluster composed of multiple servers.

In this embodiment, the application server 1 may include, but is not limited to, the memory 11, the processor 12, and the network interface 13 being communicably connected to each other through a system bus.

The application server 1 connects to the network through the network interface 13 to obtain information. The network may be an intranet, an Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, or a 5G network. Wireless or wired networks such as networks, Bluetooth, Wi-Fi, and call networks.

It is to be noted that Figure 1 shows only the application server 1 with components 11-13, but it should be understood that not all illustrated components may be implemented and that more or fewer components may be implemented instead.

The memory 11 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), and a random access memory (RAM). , static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. In some embodiments, the memory 11 may be an internal storage unit of the application server 1, such as a hard disk or memory of the application server 1. In other embodiments, the memory 11 may also be an external storage device of the application server 1, such as a plug-in hard disk equipped with the application server 1, a smart memory card (SMC), and a secure digital ( Secure Digital, SD) cards, flash cards, etc. Of course, the memory 11 can also include both the internal storage unit of the application server 1 and its external storage device. In this embodiment, the memory 11 is generally used to store an operating system installed on the application server 1 and various types of application software, such as program code of the crawler data-based user behavior analysis program 200. Further, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.

The processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 12 is typically used to control the overall operation of the application server 1, such as performing data interaction or communication related control and processing, and the like. In this embodiment, the processor 12 is configured to run program code or process data stored in the memory 11, such as running the crawler data-based user behavior analysis program 200 and the like.

The network interface 13 may comprise a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the application server 1 and other electronic devices.

In this embodiment, the user behavior analysis program 200 based on the crawler data is installed and run in the application server 1. When the crawler data-based user behavior analysis program 200 is run, the application server 1 firstly configures the crawler in advance. An application; secondly, detecting whether the crawler application is turned on; then, after opening the crawler application, again, using the crawler application to obtain crawler information; then, analyzing the crawler information and extracting keywords; Further, the user is tagged by the keyword; finally, user behavior statistics are performed according to the tag. In this way, the large amount of data acquired by the big data in the prior art can be avoided, and the preliminary range delineation of the data cannot be performed, thereby affecting the drawbacks of the accuracy of subsequent user behavior analysis. By preliminarily delineating the scope of user data, and simultaneously tagging user data in cluster processing, obtaining user behavior, improving the accuracy of data analysis, contributing to business decisions and improving marketing efficiency.

So far, the application environment of the various embodiments of the present application and the hardware structure and functions of related devices have been described in detail. Hereinafter, various embodiments of the present application will be proposed based on the above-described application environment and related devices.

First, the present application proposes a user behavior analysis program 200 based on crawler data.

Referring to FIG. 2, it is a program module diagram of the first embodiment of the user behavior analysis program 200 based on the crawler data of the present application.

In this embodiment, the crawler data-based user behavior analysis program 200 includes a series of computer program instructions stored in the memory 11, and when the computer program instructions are executed by the processor 12, the embodiments of the present application may be implemented. Analytical operations based on user behavior of crawler data. In some embodiments, the crawler data based user behavior analysis program 200 can be divided into one or more modules based on the particular operations implemented by the various portions of the computer program instructions. For example, in FIG. 2, the crawler data-based user behavior analysis program 200 can be divided into a pre-configuration module 201, a detection module 202, an acquisition module 203, an analysis extraction module 204, a label module 205, and a statistics module 206. among them:

The pre-configuration module 201 is configured to pre-configure a crawler application. Specifically, the pre-configuration module 201 configures a crawl rule of the crawler application by preset the crawler application setting window.

In this embodiment, the crawling rule includes a crawled website, content, a crawled file format (text, picture, audio, video), frequency of crawling, and the like, that is, the pre-configuration module 201 can By configuring the crawler application, the big data information of the specified scope is obtained, thereby improving the accuracy of subsequent user behavior analysis.

The detecting module 202 is configured to detect whether the crawler application is enabled.

The obtaining module 203 is configured to obtain crawler information by using the crawler application after the crawler application is started. Specifically, the crawler information includes a combination of one or more of a website crawled by the user, a file format crawled by the user, a content crawled by the user, a user's access IP, a user crawl time, and a user crawl time.

In this embodiment, the crawler application can collect a lot of crawler information. such as:

First, information about personnel registration: nickname, phone number, company name, registered address, registered IP address

Second, crawl rule information: crawled website, content, crawled file format (text, picture, audio, video), frequency of crawling;

Third, crawling data extraction information: time, extraction amount, and extraction range of data extracted by the user.

The analysis extraction module 204 is configured to analyze the crawler information and extract keywords. In this embodiment, the keyword includes a combination of one or more of a website address, a website type, a crawl content property, a format of a crawl file, an IP, a crawl time period, a crawl frequency, and a crawl amount. .

In this embodiment, if there is website information in the crawl information, the website address or the type of the corresponding website may be extracted as a keyword; if the crawl information has the user's access IP, the IP may be used as a keyword; if crawling The information is the crawl time, then the time range of a certain range is used as the keyword; if the crawling information is the content crawled by the user, the nature of the content is used as a keyword, such as video, music, picture, etc.; If the crawl information is the file format of the crawl, the corresponding format is the keyword; if the crawl information is the number of crawls, the crawl frequency or the number of crawls can be used as the keyword.

The tag module 205 is configured to tag the user by using the keyword. For example, when the URL is extracted or the type of the corresponding website is used as a keyword, the user tag can be divided into an investor, a depositor, or an insured according to the URL accessed by the user. When the keyword is IP, the user tag can be classified into schools according to the IP, and the hospital. , banks, etc., when the keywords are video, music, pictures, user tags can be divided into musicians, screenwriters, fans, etc. according to the frequency of access to video, music, and pictures.

The statistic module 206 is configured to perform user behavior statistics on the tags that are tagged by the user according to the tag module 205.

Specifically, the label module 205 collects statistics on user behaviors by using the following manners: the label module 205 clusters users who have the labels according to preset indicators, and collects user behaviors according to the collected data of the users. .

Specifically, the preset indicator includes: a proximity, a frequency, and a quota.

In this embodiment, the user behavior is also counted as a User Profile. The bottom layer of the user image is machine learning. Then, whether it is to do customer grouping or precision marketing, the user data must be processed regularly. Converted into feature vectors of the same dimension, many gorgeous algorithms can be useful, so-called regular processing such as clustering, regression, correlation, various classifiers and so on.

In this embodiment, the user is first tagged by the above scheme to define multiple dimensions. Then count the number of users based on the same dimension to analyze user behavior. In this embodiment, the recency in the preset indicator refers to the first or last crawl time of the user; the frequency refers to the user's crawl frequency; the amount (Monetory) ) refers to the number of crawls by the user. In other embodiments, other indicators may also be selected as clustering indicators. For example, the crawling information refers to the rating content, then the label may be a rating, the clustering indicator may select a crawling content rating, or the number of users may be selected as a clustering indicator. Wait.

By clustering the tagged data as described above, it is possible to obtain some websites or contents with high user attention, and also to obtain distribution of users who use the crawler program. According to the user behavior obtained by the above clustering, it can assist business decision-making and improve marketing efficiency.

Through the above program modules 201-206, the crawler data-based user behavior analysis program 200 proposed by the present application firstly pre-configures the crawler application; secondly, detects whether the crawler application is turned on; and then, opens the crawler After the application, again, using the crawler application to obtain crawler information; then, analyzing the crawler information and extracting keywords; further, using the keyword to tag the user; and finally, performing user behavior according to the tag statistics. In this way, the large amount of data acquired by the big data in the prior art can be avoided, and the preliminary range delineation of the data cannot be performed, thereby affecting the drawbacks of the accuracy of subsequent user behavior analysis. By preliminarily delineating the scope of user data, and simultaneously tagging user data in cluster processing, obtaining user behavior, improving the accuracy of data analysis, contributing to business decisions and improving marketing efficiency.

In addition, the present application also proposes a user behavior analysis method based on crawler data.

Referring to FIG. 3, it is a flowchart of the first embodiment of the user behavior analysis method based on crawler data in the present application. In this embodiment, the order of execution of the steps in the flowchart shown in FIG. 3 may be changed according to different requirements, and some steps may be omitted.

In step S301, the crawler application is pre-configured. Specifically, the application server 1 configures a crawl rule of the crawler application by preset the crawler application setting window.

In this embodiment, the crawling rule includes a crawled website, content, a crawled file format (text, picture, audio, video), frequency of crawling, and the like, that is, the application server 1 can pass Configure the crawler application to obtain the specified range of big data information, thereby improving the accuracy of subsequent user behavior analysis.

Step S302, detecting whether the crawler application is enabled. When the crawler application is started, step S303 is performed, otherwise, the flow is ended.

Step S303, acquiring crawler information by using the crawler application.

Specifically, the crawler information includes a combination of one or more of a website crawled by the user, a file format crawled by the user, a content crawled by the user, a user's access IP, a user crawl time, and a user crawl time.

First, information about personnel registration: nickname, phone number, company name, registered address, registered IP address.

Second, crawl rule information: the crawled website, content, crawled file format (text, image, audio, video), the frequency of crawling.

Step S304, analyzing the crawler information and extracting keywords.

In this embodiment, the keyword includes a combination of one or more of a website address, a website type, a crawl content property, a format of a crawl file, an IP, a crawl time period, a crawl frequency, and a crawl amount. .

Step S305, the user is tagged by using the keyword. For example, when the URL is extracted or the type of the corresponding website is used as a keyword, the user tag can be divided into an investor, a depositor, or an insured according to the URL accessed by the user. When the keyword is IP, the user tag can be classified into schools according to the IP, and the hospital. , banks, etc., when the keywords are video, music, pictures, user tags can be divided into musicians, screenwriters, fans, etc. according to the frequency of access to video, music, and pictures.

Step S306, performing user behavior statistics according to the tag. Specifically, the specific step of performing user behavior statistics according to the label is described in detail in the second embodiment (FIG. 4) of the user behavior analysis method based on the crawler data in the present application.

Through the above steps S301-306, the crawler data-based user behavior analysis method proposed by the present application, the application server 1 firstly pre-configures the crawler application; secondly, detecting whether the crawler application is turned on; and then, turning on After the crawler application, again, using the crawler application to acquire crawler information; then, analyzing the crawler information and extracting keywords; further, using the keyword to tag the user; and finally, according to the tag Perform user behavior statistics. In this way, the large amount of data acquired by the big data in the prior art can be avoided, and the preliminary range delineation of the data cannot be performed, thereby affecting the drawbacks of the accuracy of subsequent user behavior analysis. By preliminarily delineating the scope of user data, and simultaneously tagging user data in cluster processing, obtaining user behavior, improving the accuracy of data analysis, contributing to business decisions and improving marketing efficiency.

Referring to FIG. 4, it is a flowchart of a second embodiment of a user behavior analysis method based on crawler data in the present application. In this embodiment, the order of execution of the steps in the flowchart shown in FIG. 4 may be changed according to different requirements, and some steps may be omitted.

In this embodiment, the step of performing user behavior statistics according to the label specifically includes:

Step S401: Cluster the users who are tagged according to the preset indicator.

In this embodiment, the application server 1 tags the user by using the keyword, including the website address, the website type, the nature of the crawled content, the format of the crawled file, the IP, the crawling time period, and the crawling. Take a combination of one or more of the frequency and the number of crawls.

For example, when the URL is extracted or the type of the corresponding website is used as a keyword, the user tag can be divided into an investor, a depositor, or an insured according to the URL accessed by the user. When the keyword is IP, the user tag can be classified into schools according to the IP. , hospitals, banks, etc., when the keywords are video, music, pictures, user tags can be divided into musicians, screenwriters, fans, etc. according to the frequency of access to video, music, and pictures. Specifically, the preset indicator includes: a proximity, a frequency, and a quota.

Step S402, the user behavior is calculated according to the data of the clustered user. In this embodiment, the user behavior is also counted as a User Profile. The bottom layer of the user image is machine learning. Then, whether it is to do customer grouping or precision marketing, the user data must be processed regularly. Converted into feature vectors of the same dimension, many gorgeous algorithms can be useful, so-called regular processing such as clustering, regression, correlation, various classifiers and so on.

In this embodiment, through the above solution, the user is first tagged to define multiple dimensions. Then count the number of users based on the same dimension to analyze user behavior. In this embodiment, the recency in the preset indicator refers to the first or last crawl time of the user; the frequency refers to the user's crawl frequency; the amount (Monetory) ) refers to the number of crawls by the user. In other embodiments, other indicators may also be selected as clustering indicators. For example, the crawling information refers to the rating content, then the label may be a rating, the clustering indicator may select a crawling content rating, or the number of users may be selected as a clustering indicator. Wait.

Through the above steps S401-402, the user behavior analysis method based on the crawler data proposed by the present application can obtain some websites or contents with high user attention by clustering the tagged data, and can also obtain the use of the crawler program. User distribution, etc. Moreover, based on the user behavior derived from clustering, it can assist business decisions and improve marketing efficiency. .

The serial numbers of the embodiments of the present application are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is better. Implementation. Based on such understanding, the technical solution of the present application, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, The optical disc includes a number of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present application.

The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the patent application, and the equivalent structural transformation, or direct/indirect use, of the present application and the contents of the drawings is used in the application of the present application. All other related technical fields are included in the patent protection scope of the present application.

Claims

A user behavior analysis method based on crawler data is applied to an application server, and the method includes:

Pre-configure the crawler application;

Detecting whether the crawler application is enabled;

After the crawler application is opened, the crawler application is used to obtain crawler information;

Analyzing the crawler information and extracting keywords;

Using the keywords to tag users; and

User behavior statistics are performed based on the tags.
The method for analyzing user behavior based on the crawler data according to claim 1, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
The crawler data-based user behavior analysis method according to claim 2, wherein the preset indicator comprises: a proximity, a frequency, and a quota.
The reptile data-based user behavior analysis method according to claim 1, wherein the step of pre-configuring the crawler program comprises:

The crawling rules of the crawler application are configured by presetting the crawler application settings window.
The reptile data-based user behavior analysis method according to claim 4, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
The crawler data-based user behavior analysis method according to claim 1, wherein the crawler information comprises a website crawled by a user, a file format crawled by the user, a content crawled by the user, a user access IP, and a user. A combination of one or more of crawl time and user crawl times.
The reptile data-based user behavior analysis method according to claim 6, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
The crawler data-based user behavior analysis method according to claim 1, wherein the keyword includes a website address, a website type, a crawl content property, a format of a crawl file, an IP, a crawl time period, and a crawl. Take a combination of one or more of the frequency and the number of crawls.
The reptile data-based user behavior analysis method according to claim 8, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
An application server, comprising: a memory, a processor, wherein the memory stores a crawler data-based user behavior analysis program executable on the processor, the crawler data-based user The behavior analysis program is implemented by the processor to implement the following steps:

Pre-configure the crawler application;

Detecting whether the crawler application is enabled;

After the crawler application is opened, the crawler application is used to obtain crawler information;

Analyzing the crawler information and extracting keywords;

Using the keywords to tag users; and

User behavior statistics are performed based on the tags.
The application server according to claim 10, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
The application server according to claim 11, wherein the preset indicator comprises: proximity, frequency, and credit.
The application server according to claim 10, wherein the step of pre-configuring the crawler program comprises:

The crawling rules of the crawler application are configured by presetting the crawler application settings window.
The application server according to claim 13, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
The application server according to claim 10, wherein the crawler information comprises a website crawled by the user, a file format crawled by the user, a content crawled by the user, a user's access IP, a user crawl time, and a user crawl. A combination of one or more of the number of times taken.
The application server according to claim 15, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
The application server according to claim 10, wherein the keywords include a website address, a website type, a crawl content property, a format of a crawl file, an IP, a crawl time period, a crawl frequency, and a crawl amount. a combination of one or more of them.
The application server according to claim 17, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.
A computer readable storage medium, characterized in that the computer readable storage medium stores a user behavior analysis program based on crawler data, and the crawler data based user behavior analysis program can be executed by at least one processor to enable The at least one processor performs the following steps:

Pre-configure the crawler application;

Detecting whether the crawler application is enabled;

After the crawler application is opened, the crawler application is used to obtain crawler information;

Analyzing the crawler information and extracting keywords;

Using the keywords to tag users; and

User behavior statistics are performed based on the tags.
The computer readable storage medium according to claim 19, wherein the step of performing user behavior statistics according to the label comprises:

Clustering users who tag the tags according to preset criteria; and

User behavior is counted based on the data of the clustered users.