CN107832333B

CN107832333B - Method and system for constructing user network data fingerprint based on distributed processing and DPI data

Info

Publication number: CN107832333B
Application number: CN201710911155.4A
Authority: CN
Inventors: 禹可; 吴晓非; 吴楚婷; 谭尧文
Original assignee: Beijing University of Posts and Telecommunications
Current assignee: Beijing University of Posts and Telecommunications
Priority date: 2017-09-29
Filing date: 2017-09-29
Publication date: 2022-05-10
Anticipated expiration: 2037-09-29
Also published as: CN107832333A

Abstract

The invention discloses a method and a system for constructing user network data fingerprints based on a distributed processing framework and DPI data, which are used for extracting the internet access behavior characteristics and preferences of a user mobile terminal. The system comprises: a data preprocessing module: cleaning and redundancy removal are carried out on original data facing to the requirement of network data fingerprints; a rule extraction module: selecting M commonly used mobile phone APPs, capturing packets to obtain the domain name of each APP and performing regular matching, and taking the matching formula as the identification rule of each APP and forming a rule file; a user set extraction module: extracting a user set to be counted by a network data fingerprint system; the user behavior extraction module: counting the access conditions of the user to the M APPs in each unit time period; a data storage module: and storing the result partition in a data warehouse, creating an index and backing up. According to the invention, the corresponding relation between the network space and the real life is established by describing the internet surfing behavior of the mobile terminal of the user, so that convenience is provided for analyzing the user behavior of the mobile internet, and the space and time resources are saved.

Description

Method and system for constructing user network data fingerprint based on distributed processing and DPI data

Technical Field

The invention discloses a method and a system for constructing user network data fingerprints based on a distributed processing framework and DPI data, which are used for extracting behavior characteristics and preference of a user on surfing the Internet at a mobile terminal.

Background

By analyzing and extracting the characteristics of data such as network access records and the like, the data characteristics and the mode rules with remarkable markedness and distinguishing degree are obtained, and based on the data characteristics and the mode rules, a network personality and behavior research system is established, and the mode is called as data fingerprint. In view of the background of mass mobile internet data, the invention can establish the corresponding relation between the network space and the real life by accumulating and researching the network data fingerprints based on the processing mode of a distributed framework, clearly describes the behavior of the user mobile terminal network access, provides great convenience for the analysis of the user mobile terminal network behavior, and saves a large amount of storage space and resources of running time.

The network data fingerprint system provides great convenience for analyzing the network behavior of the mobile terminal of the user. The internet surfing behavior preference of the user is fully known, and the method has extremely high economic value and significance for operators and APP owners. Based on two conditions that the mobile internet user base number is large and the operator data is accurate and complete data fingerprint information, the behavior of the user can be extracted through a big data analysis and data mining method, and the track and preference of the user for surfing the internet are obtained. Through analyzing the internet surfing behavior of the user, the internet surfing requirements and preferences of the user can be better known, and then the similar APP with the competitive relationship can be found, so that the APP attribute of the user can be optimized, and the network can be adjusted.

The network data fingerprint system saves a large amount of storage space and runtime resources. The DPI data quantity generated by a telecommunication operator every day is huge, and the sending/receiving information of each data packet is simply recorded, so that a lot of effective information is buried in massive data, fields in DPI original data are redundant, and time resources and space resources in the data analysis process are seriously consumed by excessive fields. Therefore, reasonable processing and integration of the data are very necessary, on one hand, DPI data can more intuitively reflect the internet surfing track of a user, on the other hand, time and space resources can be greatly saved, and analysis and mining of the internet surfing behavior pattern of the user are facilitated. The data is stored in the database, so that mass data can be stored more safely and effectively, and the next inquiry and processing are facilitated.

Disclosure of Invention

The invention discloses a method and a system for constructing user network data fingerprints based on a distributed processing framework and DPI data from a telecom operator, which are used for extracting behavior characteristics and preference of a user on the internet at a mobile terminal.

The technical scheme adopted by the invention comprises the following five modules:

(1) a data preprocessing module: cleaning and redundancy removal are carried out on original data facing to the requirement of network data fingerprints;

(2) a rule extraction module: selecting M commonly used mobile phone APPs, capturing packets to obtain the domain name of each APP and performing regular matching, and taking a matching formula as an identification rule of each APP and forming a flow rule file;

(3) a user set extraction module: extracting a user set to be counted by a network data fingerprint system;

(4) the user behavior extraction module: counting the access condition of the user to M APPs in each unit time period

(5) A data storage module: and storing the result partition in a data warehouse, creating an index and backing up.

The network data fingerprint system filters and screens records in original DPI data in a data preprocessing module according to fields required by the network data fingerprint system, effective fields necessary for the system are finally reserved, and processing results are stored as preprocessing files for subsequent use.

In the rule extraction module, the network data fingerprint system filters out the APP with the front usage amount according to the APP usage amount condition in the preprocessed data and makes an APP list, extracts the regular expression of the domain name corresponding to each APP, the regular expression is the matching rule of the APP, the regular expressions are used for matching DPI data to verify the accuracy of the APP, and if the regular expressions are accurate, the APP and the corresponding domain name are used as the matching rule to make a matching file.

In the user set extracting module, the network data fingerprint system may adopt, but is not limited to, two methods of extracting an active user with a high number of times of accessing an APP within a period of time and a user who has accessed a certain APP within a period of time, and specifically which method is adopted to extract a user list depends on which type of user's internet behavior needs to be analyzed by the constructed network data fingerprint system.

In the network data fingerprint system, in the user behavior extraction module, each record is grouped according to the user and the internet access time period, the domain names accessed in each group are counted, and each counting result is stored as independent data.

The method has the following beneficial effects:

(1) the behavior characteristics of the mobile internet of the user are described more clearly and concisely, and the corresponding relation between the network space and the real life is established. The rules and the characteristics of the network data fingerprints are closely related to people in reality, and the network data fingerprint system can describe the internet surfing behavior information of the user more intuitively and clearly, so that a solid foundation is laid for further finding the internet surfing behavior preference of the user and mining the corresponding relation between the network behavior of the user and the characteristics of people in reality.

(2) The network data fingerprint system consumes less resources, and the processed data greatly saves time and space resources. The behavior pattern of the APP browsed by the user at the mobile terminal is extracted from the redundant original operator data, and a large amount of time resources and space resources in the post-processing process are saved through relatively low calculation consumption.

(3) The network data fingerprint system method has strong portability, can continuously update and add the internet access behavior information of the user, and keeps the continuous effectiveness of the data. The system can work for a long time only by regularly maintaining the flow rule file and ensuring the real-time effectiveness of each rule.

(4) The sorted network fingerprint data is stored in a data warehouse, and operations such as partitioning, index creation and the like are performed according to the content form of the network data fingerprint and the query requirement, so that the functions of persistently storing massive data, quickly searching the operation and the like are realized, and reasonable data backup enables the robustness and the safety of the system to be higher so as to cope with the unexpected situations such as failure of a storage space and the like.

Drawings

FIG. 1: a flow diagram of a network data fingerprint system;

FIG. 2: a flow diagram of a data preprocessing module;

FIG. 3: a flow diagram of a rule extraction module;

FIG. 4: creating a flow schematic diagram of an APP list used by a network data fingerprint system;

FIG. 5: and the flow diagram of the user behavior extraction module.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, fig. 1 is a schematic overall flow chart of a network data fingerprint system, and the method may include:

preprocessing the original data facing the requirement of the network data fingerprint, and cleaning and removing redundancy of the original data;

selecting M commonly used mobile phone APPs, obtaining a domain name of each APP through packet capturing, analyzing and regularly matching the domain name, using a matched regular expression as an identification rule of each APP, and numbering to form a flow rule file; the accurate and effective flow rule file provides important guarantee for later construction of a network data fingerprint system.

Methods for determining a set of users whose network data fingerprints need to be counted, including but not limited to methods of intercepting active users and filtering through a designated APP;

matching identification rules, and counting the access conditions of the user to the M APPs in each unit time period;

and storing the processed network data fingerprint result into a database.

Referring to fig. 2, fig. 2 is a schematic flow chart of the data preprocessing module.

In the embodiment, the functions of preprocessing the original data, cleaning and removing redundancy of the original data are realized by the requirement facing to the network data fingerprints. The specific scheme is realized by the following steps:

1) reading in original DPI data one by one, carrying out segmentation operation on each data record according to a set separator, and judging a segmentation result to determine whether the record is effective or not;

2) if the number of the separated fields is equal to the specified number of the fields, the data is valid, and the processing is continued; if the number of the separated fields is not equal to the specified number of the fields, the data is invalid, and the next piece of data is processed;

3) and similarly, filtering out records with fields in certain positions not meeting the requirement of the character type so as to filter out data with possibly disordered codes. Processing data meeting the requirements, and extracting fields required by a network data fingerprint system, wherein the fields comprise user information, user internet access APP information, user internet access time information and the like, so that a recorded effective field is extracted;

4) and storing the field, continuously reading the next file record, repeating the steps until all the original data are processed, and storing the result into the file to be used as the preprocessed file for subsequent use.

Referring to fig. 3, fig. 3 is a schematic flow chart of the rule extraction module.

In the embodiment, screening of the APP is realized, and the function of extracting the corresponding APP domain name rule file is realized. The specific scheme is realized by the following steps:

1) creating an APP list for the network data fingerprinting system, with reference to fig. 4, the specific process includes:

reading in a preprocessed file, counting the access quantity of all domain names appearing in the file within a period of time, selecting APPs with domain name access quantity in the front M as matched APP lists for constructing a network data fingerprint system, wherein the domain names corresponding to some APPs are not unique, so that reverse verification is needed according to the result of counting the domain names, accumulating the number of the domain names from the same APP, and intercepting the front M matched APP lists as network data fingerprints;

numbering the APPs in the list according to categories, wherein the numbering is the unique identifier in the APP flow rule file, and the numbering is used for describing the APPs by a simpler method;

2) performing packet grabbing processing on the first M APPs to obtain the domain name of each APP;

3) performing regular matching on the domain name obtained by packet capturing, extracting a domain name regular expression and taking the domain name regular expression as a matching rule of the domain name, so as to achieve the purpose of processing, namely, verifying the domain name accuracy of DPI data, and ensuring that all effective domain names of each APP are extracted, so that a network data fingerprint system is more perfect and comprehensive;

4) matching a part of DPI data by using a regular matching formula to prove the accuracy of the extracted regular matching formula;

5) and sorting the numbers, the domain names and the regular expressions to enable the numbers, the domain names and the regular expressions to be in one-to-one correspondence, and finally forming a rule matching file.

The fields of the rule matching file include:

APP _ rule _ code: uniquely identifying each domain name;

name: the APP name;

category: the APP category;

domain: the domain name of each APP is the matching rule, and the domain name corresponding to each APP is not unique.

And a user set extraction module: a method for determining a user set to be counted by the network data fingerprint;

the DPI data from the operator records network records of all users receiving the service of the operator, and in view of the situations that some users have low online activity and lack of directivity, the preference of network behaviors of the users is not obvious, and the like, the module selects a part of users as construction objects of network data fingerprints, and then analyzes the network behavior characteristics of the part of users.

The method for filtering out the interested user set includes, but is not limited to, the following two methods, specifically which filtering method is adopted, depending on which part of users are more interested:

firstly, from the perspective of the internet activity, the more active users leave more records in the network behavior, and the more easily the users find the network behavior characteristics and preferences.

The method is implemented by the following specific steps: reading a pretreatment file for a period of time, grouping data according to users, namely grouping all access data generated by the same user into a group, counting the access number of APP in the file of each user flow measurement rule, comparing the access number among the users, and finally selecting the user with larger access number as a user list of network data fingerprints to be stored in a user list file.

And secondly, starting from a certain APP, counting all users accessing the certain APP, and taking the users as a user set, so as to observe whether the users accessing the certain APP have other behavior commonalities, and the directivity is more definite.

The method is implemented by the following specific steps: and selecting one APP as the APP for filtering, and finding out the corresponding matching rule according to the flow rule file to filter the user. Reading the preprocessing files for a period of time, checking the domain name condition accessed by each user, and if the domain name accessed by one user comprises the domain name corresponding to the APP, storing the user information; otherwise, skipping the data and continuing to process the next data; if a certain user is found, the user does not need to be added again, and finally all screened user lists are stored in the user list file.

Referring to fig. 5, fig. 5 is a schematic flow chart of the user behavior extraction module.

In this embodiment, the implemented function is to match the identification rule and count access conditions of the user to the M APPs in each unit time period, and the specific operation method includes:

1) reading in the preprocessed file;

2) judging whether the user generating the record is in a user list or not, and if the user is in the list, performing the next operation; if not, discarding the data and continuing to process the next data;

3) processing the filtered data, and grouping according to the combined action of the users and the time periods, namely grouping the information of the same users and the same time periods into a group;

4) the processing method for a group of user time slots is as follows: traversing each domain name, judging whether the domain name is in a flow rule file, if so, proving that a user accesses an APP in an APP list, adding one to the corresponding accessed APP number, and continuing to process the next domain name, otherwise, proving that the user does not access the APP in the APP list, discarding the data, and continuing to process the next domain name until all the domain names in the group are processed, which indicates that the processing of the user time period group is finished.

5) Processing other groups continuously according to the method until all groups are processed, and obtaining the result that the network data fingerprint is obtained

Processing the result, wherein the fields comprise:

id: a label uniquely identifying each record;

meid: a terminal device number of the user;

rule _ code: an encoding corresponding to the flow rule file;

pv: the operation click rate (page browsing amount) of the APP by the user;

provice: the province of the user;

report _ date: the occurrence date of the user internet behavior;

hour _ period: the occurrence time period of the user internet behavior;

create _ date: the creation time is recorded.

A data saving module: and storing the processed network data fingerprint results into a database one by one, wherein the specific operation flow comprises the following steps:

1) creating a storage table in a data warehouse, creating a table header according to a statistical field of the network data fingerprint, and partitioning the data warehouse by day in view of the condition that the data volume of a network data fingerprint system in one day is extremely large;

2) storing the content of the network data fingerprint processed based on the distributed architecture into a corresponding partition;

3) in order to improve the query efficiency and enhance the pertinence, and according to the purpose of analyzing the behavior and the preference of a user mobile internet terminal, hash indexes are respectively established according to three fields of an APP name (domain), a user and a recording time period so as to meet the diversified query requirements.

4) In order to ensure the safety and the soundness of the system, the system performs backup processing on data, and each segment needs to store two parts of data, including:

data of a day in a network fingerprint system;

as a backup, the data of the day in the previous segment is stored, wherein the data of the last segment is backed up in the first segment.

5) And storing corresponding storage information and backup information in the host.

Claims

1. A method for constructing a user network data fingerprint based on distributed processing and DPI data is characterized by comprising the following steps:

preprocessing original DPI data by a data preprocessing module facing to the requirement of network data fingerprints, wherein the preprocessing comprises cleaning and redundancy removal;

selecting M commonly used mobile phone APPs through a rule extraction module, capturing packets to obtain a domain name of each APP and performing regular matching, and taking a matching formula as an identification rule of each APP to form a flow rule file;

extracting a user set to be counted by a network data fingerprint system through a user set extraction module;

matching identification rules through a user behavior extraction module, and counting the access conditions of a user to M APPs in each unit time period to obtain a processing result of the network data fingerprint;

storing the processing result partitions of the network data fingerprints into a data warehouse through a data storage module, and creating and backing up indexes;

select M commonly used cell-phone APP through rule extraction module, grab the package and obtain every APP's domain name and regular match, regard the identification rule of every APP and form flow rule file with the matching formula, include:

ranking the statistical results of the traffic data corpus, and intercepting the first M APPs to prepare an APP list of network data fingerprints;

numbering M APPs according to categories, wherein the numbering is the unique identifier in the APP flow rule file; and

sorting the matched domain names to form flow rule files, wherein the flow rule files correspond to the numbers;

through the matching recognition rule of the user behavior extraction module, the access condition of the user to M APPs in each unit time period is counted, and the method comprises the following steps:

based on a distributed processing framework, processing the preprocessed data, and searching the network access behavior of each user;

calculating the access number of the user to the APP needing to be counted in each unit time period; and

the number of accesses per APP per unit time period per user is designed as one record.

2. The method of claim 1, wherein the cleansing and redundancy removal of the raw data based on the requirement of the network data fingerprint comprises:

deleting DPI records with the length of the total data field not meeting the requirement;

deleting DPI records of which the critical fields do not meet the requirements of the data types; and

each DPI record is processed, leaving the fields that need to be further processed.

3. The method of claim 1, wherein the extracting the user set of the network data fingerprint system needing statistics by the extracting user set module comprises:

counting users with the number of internet clicks larger than a certain threshold in original DPI data, and formulating a user list;

all users accessing a certain specified APP are collected, and a user list is formulated.

4. The method of claim 3, wherein the identifying rule is matched through a user behavior extracting module, and the access condition of the user to the M APPs in each unit time period is counted, further comprising:

if the user does not access an APP within a certain period of time, the current data needs to be filtered out, that is, the record is not recorded.

5. The method of claim 4, the data that needs to be filtered out comprising:

data for which the user is empty or for which the user is not in the user list;

the domain name is null or data for which the domain name is not in the regular file list.

6. The method of claim 1 or 5, wherein saving the result partition to the data warehouse via the data storage module, creating an index and backing up comprises:

carrying out partition processing on the data warehouse;

storing the content of the network data fingerprint processed based on the distributed architecture into a data warehouse;

creating a hash index according to different key values so as to quickly query data;

and backing up the data of each partition in different storage spaces so as to improve the robustness and the safety of the system.

7. The method of claim 6, wherein the data warehouse is partitioned to make the data storage format clearer and the query more rapid, and the data warehouse is stored in the data warehouse, the method comprising:

and taking the occurrence date of each record of the network data fingerprint as the basis of the partition, namely storing the data result into the corresponding partition according to the occurrence date.

8. The method of claim 6, wherein the creating a hash index for the partitioned result stored in the database is required to improve efficiency of querying the data, and comprises:

creating a hash index by taking the APP domain name as a key value;

creating a hash index by taking a user as a key value;

and taking the occurrence time period of each record as a key value, and creating a hash index.