WO2018177167A1

WO2018177167A1 - Method for analyzing ip address, system, computer readable storage medium, and computer device

Info

Publication number: WO2018177167A1
Application number: PCT/CN2018/079732
Authority: WO
Inventors: 刘鑫琪; 童剑
Original assignee: 贵州白山云科技有限公司
Priority date: 2017-04-01
Filing date: 2018-03-21
Publication date: 2018-10-04
Also published as: CN107707516B; CN107707516A

Abstract

Provided are a method for analyzing an IP address, a system, a computer readable storage medium, and a computer device. The method comprises: collecting historical data of an IP address; and performing analysis on the historical data of the IP address to generate reliability data of the IP address. The present invention is applicable to aspects such as legitimacy authentication of an IP address and interception of an IP address, thereby solving a problem in which a security class of an IP address is mistakenly identified, effectively preventing misjudgment of legitimacy of an IP address, and preventing an erroneous interception of an IP address.

Description

IP address analysis method, system and computer readable storage medium and computer equipment

The present application claims priority to Chinese Patent Application No. 200910216069.1, entitled "A Method for the Analysis of IP Addresses and Systems", filed on April 1, 2017, the entire contents of which is incorporated herein by reference. in.

Technical field

Embodiments of the present invention relate to, but are not limited to, the field of Internet user portraits, and in particular, to an IP address analysis method, system, and computer readable storage medium and computer device.

Background technique

The network layer firewall can be thought of as an IP packet filter that operates on the underlying TCP/IP protocol stack. In the enumeration mode, only the packets that meet the specific rules are allowed to pass, and the rest are prohibited from crossing the firewall (except for the virus, the firewall cannot prevent the virus from intruding). These rules can usually be defined or modified by the administrator, although some firewall devices may only be able to apply built-in rules.

Firewall rules can also be formulated in a more relaxed way, as long as the packet does not conform to any of the "negative rules". Most operating systems and network devices have built-in firewall functions.

Newer firewalls can use the various attributes of the packet to filter, such as source IP address, source port number, destination IP address or port number, and service type (such as HTTP or FTP). It can also be filtered via attributes such as communication protocol, TTL value, source domain name or network segment....

The existing interception scheme is intercepted by a single fixed rule, and the granularity and dimension of the access request analysis are insufficient, and the identity of the visitor is not recognized, which is likely to cause a mistake.

Summary of the invention

Embodiments of the present invention are directed to solving the problems described above. It is an object of embodiments of the present invention to provide a solution to any of the above problems. Specifically, embodiments of the present invention provide the capability.

According to a first aspect of the embodiments of the present invention, an IP address analysis method is provided, including:

Collect historical data of IP addresses;

The historical data of the IP address is analyzed to generate credit data of the IP address.

The step of collecting historical data of the IP address includes:

Collect and parse the original log in the first cycle;

Formatting the information in the original log to obtain a predefined indicator, the predefined indicator including at least one or more of the following information:

Time, IP, working time request number, rest period request number, sleep period request number, working period request file size, rest period request file size, sleep period request file size, working period user agent UserAgent number, rest period UserAgent number, sleep Number of time UserAgents, number of mobile UserAgents, number of UserAgents on the PC, number of access sources, number of visited domain names, number of hours of occurrence;

The predefined indicators corresponding to the respective IP addresses are stored, and each IP address corresponds to one or more predefined indicators.

The step of collecting historical data of the IP address further includes:

In the second cycle, third-party IP libraries and/or third-party IP blacklists are obtained from third-party platforms one or more times.

The historical data of the IP address is analyzed, and the credit data of the generated IP address includes:

Pre-defining the pre-defined indicators of the first period corresponding to the working days in the second period and normalizing to obtain intermediate values of the respective working days, and pre-defining the first period corresponding to the rest days in the second period The indicator is pre-processed and normalized to obtain an intermediate value of each rest day, where the second period includes a plurality of first periods corresponding to the working days and a plurality of first periods corresponding to the rest days;

Performing weighted average processing on the intermediate values of the respective working days to obtain a working day weighted average value;

Performing a weighted average or maximum value for each of the rest day intermediate values to obtain a rest day weighted mean or maximum value;

The current second period temporary specific indicator in the second period is calculated according to one or more working day weighted averages and one or more rest day weighted average values, and the current second period temporary specific indicators include:

This week is the probability of office export IP. This period is the probability of household export IP. This period is the probability of real person, the activity score of this cycle, and the number of people in this cycle.

Adjusting the current second period temporary specific indicator according to the final specific indicator of the previous second period and the third-party IP library and/or the third-party IP blacklist, and obtaining the final specific indicator of the current second period, The final specific indicator of the second period serves as credit data for the IP address.

The pre-predetermined indicator of the first period corresponding to the working day in the second period is pre-processed and normalized to obtain an intermediate value of each working day, and the first corresponding to the rest day in the second period The steps of pre-processing and normalizing the periodic pre-defined indicators to obtain the intermediate values of each rest day include:

Calculate the first cycle hour number score corresponding to the working day, the mobile userAgent score, the PC userAgent score, the rest period VS work time request number score, the rest period VS sleep time request number score, the access domain name score, and the above The score is taken as a weighted mean, and the working day is the intermediate value of the household exit IP probability;

Calculating the first cycle working period request number score corresponding to the working day, the working period VS rest period request number score, the rest period VS sleep period request number score, the PC end UserAgent number score, the working period VS rest period UserAgent number score, and the above The weighted average of the points is obtained, and the working day is the intermediate value of the office exit IP probability;

Calculate the first cycle request number distribution score, the UserAgent number distribution score, the hour appearance number score, the domain name number VS source number score, the mobile terminal VS PC end UserAgent number score corresponding to the working day, and the weighted average value of the above scores is obtained, and the working day is obtained. The median probability of being a real person;

Calculating the first period access domain number score, the working period request number score, the rest period request number score, the sleep period request number score, the hour appearance number score, the request source number score corresponding to the working day, and weighting the average score of the above scores, Intermediate value of working day activity;

Calculate the first cycle hour number score corresponding to the rest day, the mobile userAgent score, the PC end UserAgent number score, the rest period VS work period request number score, the rest period VS sleep period request number score, the access domain name score, and the above The score takes the weighted mean value, and the rest day is the intermediate value of the household exit IP probability;

Calculate the first period rest period request number score, the UserAgent number distribution score, the hour appearance number score, the domain name number VS source number score, the mobile end VS PC end UserAgent number score corresponding to the rest day, and take the weighted mean value of the above scores to obtain a rest The daily value of the probability of a real person;

Calculating the first period access domain number score, the work period request number score, the rest period request number score, the sleep period request number score, the hour appearance number score, the request source number score corresponding to the rest day, and weighting the average score of the above scores, The median value of the rest day activity.

The step of calculating the current second period temporary specific indicator in the second period according to one or more working day weighted average values and one or more rest day weighted average values includes:

After pre-processing and normalization, the number of UserAgents on the workday VS rest day, the number of workdays on the workday VS rest days, the number of workdays VS rest days, and the number of breaks on the rest days are obtained. The weighted average value of the intermediate value of the office exit IP probability is the probability that the current period is the office exit IP;

The working day is the intermediate value of the household export IP probability and the rest day is the weighted average of the intermediate value of the household export IP probability as the current period is the probability of the household export IP;

The working day is the weighted mean of the median probability of the real person and the rest day as the median probability of the real person as the true probability of the cycle;

The weighted average of the median activity day value and the median value of the rest day activity is used as the cycle activity score.

The number of mobile Agents on the working day and the number of mobile Agents on the rest day and the maximum number of PCs on the workday and the number of UserAgents on the rest of the PC are grouped as the number of people in this cycle.

Wherein, the final specific indicator includes at least one or more of the following information,

IP, IPInt, update ID, the number of IP updates, the final number of people grouped, and finally the sum of the probability of office export IP, and finally the sum of the probability of household export IP, and finally the sum of the probability of real people, the sum of the final activity scores,

The IPInt is a long integer corresponding to the IP address, the Update ID is the number of times to update the final specific indicator of the second period, and the IP update number is the final specific indicator of the second period of the update of the IP address. frequency,

The step of adjusting the temporary specific indicator of the current second period according to the final specific indicator of the previous second period and the third-party IP library and/or the third-party IP blacklist to obtain the final specific indicator of the current second period include:

For the IP address involved in the final specific indicator of the last second period and the temporary specific indicator of the current second period, the final specific indicator of the current second period is obtained by the following calculation:

When the current number of people of the temporary specific indicator in the second period and the final number of the final specific indicator of the previous second period are grouped into adjacent groups, the group with the large number of people is selected as the final number of groups in the current second period, otherwise the current selection is selected. The number of people in the second cycle of temporary specific indicators,

The sum of the probability of the office exit IP in the temporary specific indicator of the second period plus the probability of the final office outlet IP in the final specific indicator of the second period is the sum of the probability of the final office IP of the current second period.

The sum of the probability of the household export IP plus the probability of the final household IP in the final specific indicator of the second period in the temporary specific indicator of the second period is the sum of the probability of the final household IP of the current second period.

The sum of the probability of the real person in the temporary specific indicator of the second period plus the final probability of the real person in the final specific indicator of the second period is the sum of the probability of the real person in the current second period,

After the sum of the activity score in the temporary specific indicator of the second period plus the final activity score of the second period, divided by the number of final updates recorded by the update ID, multiplied by the number of IP updates, as the current second period The sum of the final activity scores;

For the IP address involved in the temporary specific indicator of the current second period, which is not involved in the final specific indicator of the last second period, obtain the temporary specific indicator of the current second period by the following calculation:

Grouping the number of people in the temporary specific indicator of the current second period as the final number of people in the current second period.

Taking the IP of the office exit IP in the temporary specific indicator of the current second period as the sum of the probability of the final office IP of the current second period,

Taking the probability of household export IP in the temporary specific indicator of the current second period as the sum of the probability of the final household IP of the current second period,

Taking the probability of being a real person in the temporary specific indicator of the current second period as the sum of the final real probability of the current second period,

Dividing the activity score in the temporary specific indicator of the current second period by the update ID, multiplying the IP update count as the sum of the final activity scores of the current second period;

The final specific indicator of the current second period is used to cover the final specific indicator of the previous second period, and the number of times to update the final specific indicator of the second period and the number of updates to the corresponding IP address are recorded.

The final specific indicator of the current second period is adjusted according to the final specific indicator of the previous second period and the third-party IP library and/or the third-party IP blacklist, and the final specific indicator of the current second period is obtained. The steps also include:

Filtering data corresponding to the IP non-syntax or the corresponding IP as the local area network IP in the temporary specific indicator of the current second period;

According to the additional information of the IP address included in the third-party IP library, adjust the IP probability of the office exit, the probability of the household exit IP, and the probability of the real person in the temporary specific indicators of the current second period;

Generating IP credit smear data according to the third-party IP blacklist, and adding the IP credit smear data to the final specific indicator of the current second period.

Wherein, the method further comprises:

Providing an interface to a third party to allow access to credit data of the IP address through the interface; or

Receiving a third party to issue an IP verification request for the IP address, searching for credit data corresponding to the IP address, performing credit rating evaluation on the IP address according to the credit data, and returning the evaluation result to the third party.

According to another aspect of the embodiments of the present invention, an IP address analysis system is further provided, including a big data platform and an offline computing platform;

The big data platform is configured to store original logs, calculate original logs, and collect and store historical data of IP addresses;

The offline computing platform is configured to analyze historical data of an IP address collected by the big data platform to generate credit data of an IP address.

According to another aspect of an embodiment of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program, the program being implemented by a processor to implement the steps of the above method.

According to another aspect of an embodiment of the present invention, there is also provided a computer apparatus comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, the processor executing the The steps of the above method are implemented when the program is executed.

The method and system of the embodiment of the present invention collect historical data of an IP address, analyze historical data of an IP address, generate credit data of an IP address, implement accurate analysis of IP address refinement, and determine an IP address attribute by using big data. It has a comprehensive and accurate understanding of the IP address credit situation, and can be applied to scenarios such as IP address legality verification and IP address interception, which solves the problem of IP address security cognitive error, effectively preventing IP address legality misjudgment, The occurrence of an IP address error.

Other characteristics and advantages of the present invention will become apparent from the following description of the exemplary embodiments.

DRAWINGS

The accompanying drawings, which are incorporated in FIG In the drawings, like reference numerals are used to refer to the like. The drawings in the following description are some embodiments of the invention, rather than all embodiments. Other figures may be obtained from those of ordinary skill in the art in light of the inventive work.

FIG. 1 exemplarily shows a flow of an IP address analysis method according to Embodiment 1 of the present invention;

FIG. 2 exemplarily shows an application principle of the technical solution provided by the embodiment of the present invention;

FIG. 3 exemplarily shows an architecture of an IP address analysis system provided by Embodiment 2 of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. It is a partial embodiment of the invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention. It should be noted that, in the case of no conflict, the features in the embodiments and the embodiments in the present application may be arbitrarily combined with each other.

In order to solve the above problems, an embodiment of the present invention provides an IP address analysis method, and Embodiment 1 of the present invention will be described below with reference to the accompanying drawings.

The embodiment of the present invention provides an IP address analysis method, which performs detailed analysis on an IP address based on historical data of an IP address and related additional information, and obtains credit data of an IP address according to the credit data of the IP address. The IP address is evaluated, and the accurate analysis and judgment of the IP address refinement is realized. The security level of the IP address is evaluated by the credit data, and the occurrence of misjudgment and error prevention is effectively prevented. The specific process is as shown in FIG. 1 and includes:

Step 101: Collect historical data of an IP address.

In this step, the original log, such as the CDN server log, is collected and parsed according to the first cycle, and the information in the CDN server log is formatted to obtain a predefined indicator. The CDN server log is specifically a CDN nginx log, and can also be calculated by using other network access logs.

The predefined indicator includes at least one or more of the following information:

Time, IP, working time request number, rest period request number, sleep period request number, working period request file size, rest period request file size, sleep period request file size, working period UserAgent number, rest period UserAgent number, sleep period UserAgent Number, number of mobile userUsers, number of user agents on the PC side, number of access sources, number of access domain names, number of hours of occurrence, wherein the first period includes a working period, a rest period, and a sleep period, and the "time" refers to a corresponding The time when the CDN server logs are generated (that is, the user access time), the "IP" refers to the IP address involved, and the "working period request number" refers to the number of requests sent by the IP address during the working period. The "rest period request number" refers to the number of requests issued by the IP address during the rest period, and the "sleep period request number" refers to the number of requests issued by the IP address during the sleep period, The “working period request file size” refers to a sum of file sizes requested by the IP address during the working period, the “ The "intermediate request file size" refers to the sum of the file sizes requested by the IP address during the rest period, and the "sleep period request file size" refers to the sum of the file sizes requested by the IP address during the sleep period. The “number of UserAgents in the working period” refers to the number of UserAgents that appear under the IP address in the working period, and the “number of User Agents in the rest period” refers to the number of users that appear under the IP address during the rest period. The number of UserAgents, the number of UserAgents in the sleep period, the number of UserAgents that appear under the IP address in the sleep period, and the number of UserAgents in the mobile terminal is the number of UserAgents accessed by the mobile terminal. The number of the user-accessible IP addresses refers to the number of UserAgents accessed by the IP address, and the number of access sources refers to the number of access sources of the IP addresses in the first period. The "number of access domain names" refers to the number of domain names accessed by the IP address in the first period, and the "hours of occurrence" refers to the occurrence of the IP address in the first period. Number of hours (h appears IP access count appears 1 hours, for example 2 and 4-point access will appear both IP value is set to 2 hours).

The predefined indicators corresponding to the respective IP addresses are stored, and each IP address corresponds to one or more predefined indicators. Specifically, the predefined indicator may be stored in a hive table.

The first period involved in this step is preferably 1 natural day (24 hours).

Preferably, the historical data of the IP address can also be obtained from a third-party platform, such as obtaining a third-party IP library and/or a third-party IP blacklist from a third-party platform one or more times in the second period, and the third-party IP library often Additional information including the IP address (such as the IP address or the distribution of the IP address segment, that is, the country, province, city, and operator corresponding to the IP segment, may also indicate the name of a company, or may be marked as a data center). Third-party platform data is not updated regularly, so it can be obtained after the data of the third-party platform is updated, or it can be obtained before the second cycle is ready for calculation.

Step 102: Analyze historical data of the IP address, and generate credit data of the IP address;

In this step, the pre-defined indicator of the first period corresponding to the working day in the second period is pre-processed and normalized to obtain an intermediate value of each working day, and the first corresponding to the rest day in the second period Pre-defined indicators of the period are pre-processed and normalized to obtain intermediate values of the respective rest days, wherein the second period includes a plurality of first periods corresponding to the working days and a plurality of first periods corresponding to the rest days; The median values of each working day are respectively weighted average (including arithmetic weighted average or geometric weighted average) to obtain the working day weighted mean; and the median values of the respective rest days are respectively weighted average or maximum processed to obtain the rest day weighted mean or maximum The current second period temporary specific indicator in the second period is calculated according to one or more working day weighted averages, and one or more rest day weighted average values, and the current second period temporary specific indicators include: the current period is office The probability of exporting IP, this period is the probability of household export IP, this period is the probability of real person, the activity score of this cycle, the number of people in this cycle Adjusting the current second periodic temporary specific indicator according to the final specific indicator of the previous second period and the third-party IP library and/or the third-party IP blacklist, and obtaining the final specific indicator of the current second period, The final specific indicator of the second period serves as credit data for the IP address.

The second period is an integer multiple of the first period; preferably, when the first period is day, the second period is month or week.

The following is an example of a specific algorithm for this step. The first period involved is 1 day, and the second period is 1 month; the normalization algorithm involved in the embodiment of the present invention uses a deformed sigmoid function, 1.0/(1.0+math.exp(-molecule/denominator+4.0 )), because the input values are greater than or equal to 0, so add 4.0.

1. Summary of working day data:

a) Find the median daily data of the working day

In this step, the IP number is used as the dimension, and the number of mobile UserAgents, the number of PC-side UserAgents, and the number of requests is calculated in the first cycle corresponding to the working day.

Number of requests: The number of work time requests plus the number of break time requests plus the number of sleep time requests.

(1) The working day IP address is the probability of household export IP:

The number of hours appears: the normalization algorithm has a numerator of 6, and the denominator is the number of hours of occurrence;

Mobile UserAgent score: normalization algorithm, the numerator is the average of the number of mobile UserAgents under each IP for a period of time (such as 10, irregular update), and the denominator is the number of mobile UserAgents;

PC-side UserAgent score: normalization algorithm, the numerator is the average of the number of PC-side UserAgents under each IP for a period of time (such as 5, irregular update), and the denominator is the number of PC-side UserAgents;

Break time VS work time request number score: normalization algorithm, the number of breaks is divided by 4, the denominator is the number of work time requests divided by 12;

Rest time VS sleep time request number score: normalization algorithm, the number of breaks is divided by 4, the denominator is the number of sleep time requests divided by 8;

Access domain name score: normalization algorithm, the numerator is the number of domain names visited by IP on a daily basis (not updated regularly), and the denominator is the number of domain names;

The number of scores in the above hours, the mobile userUser score, the PC userAgent score, the rest period VS working period request score, the rest period VS sleep period request score, and the number of access domain number scores are the working days for the household export IP probability. Median.

(2) The working day is the intermediate value of the office exit IP probability:

Work time request number score: normalization algorithm, the number of work time requests divided by 12, the denominator is the average number of hours of each IP working time period within a period of time (not regularly updated);

Work time VS break time request number score: pre-processing and normalization algorithm, the number of work is the number of work time periods divided by 12, the denominator is the number of break time requests divided by 4;

Rest time VS sleep period request score: preprocessing and normalization algorithm, the number of breaks is divided by 4, the denominator is the number of sleep time requests divided by 8;

PC-side UserAgent score: pre-processing and normalization algorithm, the numerator is the average of the number of PC-side UserAgents under each IP for a period of time (such as 10, irregular update), and the denominator is the number of PC-side UserAgents;

Working period VS break period UserAgent number score: preprocessing and normalization algorithm, the numerator is the number of UserAgents during the working period, and the denominator is the number of UserAgents during the break period;

The above working hours request number score, working period VS rest period request number score, rest period VS sleep period request number score, PC side UserAgent number score, working period VS rest period UserAgent number score, weighted average of the above score values, and get the working day The intermediate value of the IP probability of the office exit;

(3) The working day is the median probability of real person:

Request number distribution score: preprocessing and normalization algorithm, the numerator is the number of work period requests divided by 12 and the number of rest periods is divided by 4 and the sleep period is divided by 8 standard deviation, the denominator is 1;

UserAgent number distribution score: normalization algorithm, the number of users is the number of UserAgents during the working period, the number of UserAgents during the break period, the standard deviation of the number of UserAgents during the sleep period, and the denominator is 1;

The number of hours appears: the normalization algorithm, the numerator is 6, and the denominator is the number of hours;

Domain name VS source number score: normalization algorithm, the numerator is the source number, and the denominator is the number of domain names;

Mobile Agent VSPC UserAgent Number Score: Normalization algorithm, the numerator is the number of mobile UserAgent, and the denominator is the number of PC UserAgent.

The above score weighted mean value is the median probability of the real person on the working day.

(4) Intermediate value of working day activity:

Number of access domain names: normalization algorithm, the number of accesses is the number of domain names, and the denominator is 10;

Break time request score: normalization algorithm, the number of breaks is divided by 4, and the denominator is the average number of hours of each IP break in a period of time (not regularly updated);

Sleep time request score: normalization algorithm, the number of sleep time requests divided by 8, the denominator is the average number of hours of IP sleep time requests within a period of time (not regularly updated);

The number of hours appears: normalization algorithm, the number of molecules is the number of hours, and the denominator is 6;

Request source score: Normalization algorithm, the numerator is the number of request sources, and the denominator is the average of the average number of daily requests per IP (not updated regularly).

All of the above score weighted mean values get the median activity day value.

b) Find the weighted mean of the median daily data on weekdays:

Taking IP as the dimension, calculate the weighted average of the number of mobile userUsers in the first cycle corresponding to the working day, the weighted mean of the number of UserAgents on the PC, the weighted average of the number of requests, the working day is the weighted average of the probability of the home exit IP, and the working day is the weight of the office exit IP probability. Mean, the working day is the real-life probability weighted mean, and the working day activity is weighted by the mean.

2. Summary of rest day data:

a) Find the median value of the daily data on the rest day

The IP is used as a dimension to calculate the number of mobile UserAgents, the number of PC-side UserAgents, and the number of requests in the first cycle corresponding to the rest day.

(1) The rest day is the intermediate value of the household export IP probability: similar to the working day algorithm;

(2) The rest day is the median probability of the real person: similar to the working day algorithm;

(3) Intermediate value of rest day activity: similar to the working day algorithm.

b) Find the weighted mean or maximum of the median daily data for the rest day:

Taking IP as the dimension, calculate the maximum number of mobile UserAgents in the first cycle corresponding to the rest day, the maximum number of UserAgents on the PC side, and the weighted mean of the number of requests. The rest day is the weighted average of the probability of household export IP, and the rest day is the weighted average of the probability of the real person. , rest day activity weighted mean.

3. The data of working days and rest days are summarized, and the current specific indicators of the second cycle are obtained:

Calculate the workdays and rest days according to the IP connection and get:

IPInt: Convert this IP to the corresponding long integer.

The number of people in this week group: Find the number of UserAgents on the PC side of the working day, and calculate the weighted average of the number of PC-side UserAgents in the first cycle corresponding to the working day. The number of mobile Agents on the working day is calculated, and the weighted average of the number of mobile userUsers in the first cycle corresponding to the working day is calculated. On the rest day, the number of UserAgents on the PC side calculates the weighted average of the number of PC-side UserAgents in the first cycle corresponding to the rest day. The number of mobile Agents on the rest day is calculated, and the weighted average of the number of mobile users in the first cycle corresponding to the rest day is calculated. After finding the maximum number of PC-side UserAgents on the working day, the number of working-time mobile-side UserAgents, the number of PC-side UserAgents on the rest day, and the number of UserAgents on the rest day, the group is grouped as follows: 1:0-1, 2:2-5, 3:6-10, 4:11-30, 5:31-50, 6:51-100, 7:101-500, 8:501-2000, 9:>2000.

This week is the probability of office export IP:

Workday VS rest day PC-side UserAgent number score: normalization algorithm, the numerator is the number of PC-side UserAgents on the working day, and the denominator is the number of PC-side UserAgents on the rest day;

Workday VS rest day Mobile UserAgent score: normalization algorithm, the number of numerators is the number of UserAgents on the working day, and the denominator is the number of UserAgents on the rest day.

Workday VS rest day request score: normalization algorithm, the numerator is the number of workday requests, and the denominator is the rest day request number;

The weighted average of the above three score weighted mean values and the median value of the office exit IP probability on the working day is the IP probability of the office exit IP in this cycle.

This week is the probability of household export IP: the working day is the weighted average of the intermediate value of the household export IP probability and the rest day is the intermediate value of the household export IP probability.

This week is the probability of a real person: the working day is the weighted mean of the median probability of the real person and the rest day is the median probability of the real person.

This week's activity score: the weighted average of the median activity day value and the median value of the rest day activity.

This week's number of people grouping: grouping the number of mobile Agents on the working day and the number of mobile Agents on the rest day and the maximum number of PCs on the workday and the number of UserAgents on the rest of the PC.

The temporary specific indicators of the current second period are stored in the MySQL temporary data table and enter the adjustment phase.

Adjustment phase

The current specific indicator of the current second cycle stored in MySQL, and the final specific indicator of the last second cycle stored in MySQL in the previous second cycle are used as input for this phase.

Traverse all the IPs in the MySQL temporary data table, each IP corresponding to the current second period temporary specific indicators.

1. Filter out IP grammatical data and filter out LAN IP.

2. Obtain third-party IP library information, determine whether the additional information contained in the third-party IP library determines whether the string contains the following sensitive string, and returns the corresponding adjustment index: “Company”, “Data Center”, “GSM/TD-SCDMA” /LTE". The adjustment index includes three adjustment indexes of “probability of real person”, “probability of IP for office export”, and “probability of IP for household export”. If no sensitive string is included, the adjustment index of all three probabilities is 1. The three probabilities are multiplied by three probabilities, and the probabilities need to be in the range of [0.05, 0.95]. If they are less than 0.05, they return 0.05. If they are greater than 0.95, they return 0.95.

3. Obtain the final specific indicator of the last second period deposited in MySQL.

For an IP address, according to whether the IP address has the final specific indicator of the previous second period, the manner in which the final specific indicator of the current second period is generated for the IP address is also different, as follows:

a) If there is a final specific indicator of the last second period of this IP, the corresponding indicators are operated accordingly:

Update ID: The data of the second cycle is updated several times.

IPInt: Convert this IP to the corresponding long integer.

The number of IP updates: The number of IP updates is increased by one in the previous second period.

Final number grouping: If the current number of people in the second period of the temporary specific indicator group and the final number of the final specific indicator in the previous second period are grouped into adjacent groups, the group is eventually large; otherwise it is the grouping of temporary data.

Finally, the sum of the IP probabilities of the office exits: the sum of the office exit IP probabilities in the current second-period temporary specific indicators plus the final concrete IP probabilities in the final specific indicators of the second period.

Finally, the sum of the probability of household export IP: the current second-period temporary specific indicator is the sum of the household export IP probability plus the final specific indicator of the second cycle, which is the sum of the household export IP probabilities.

Finally, the sum of the probabilities of the real people: the probability of the real IP in the temporary specific indicator of the second period plus the final probability of the real person in the final specific indicator of the second period.

The sum of the final activity scores: the sum of the activity scores in the temporary specific indicators of the current second period plus the final activity scores of the second period, divided by the number of final updates recorded by the update ID, multiplied by the number of IP updates .

If there is no such IP, the corresponding operations are performed on the following indicators:

IPInt: Convert this IP to the corresponding long integer.

Update ID: The data of the second cycle is updated several times.

The number of IP updates: 1.

Final grouping: The current number of people in the second cycle of temporary specific indicators.

Finally, the sum of the IP probabilities of the office exits: the temporary specific indicators of the current second period are the IP addresses of the office exits.

Finally, the sum of the probability of household export IP: the current specific indicator of the second cycle is the probability of household export IP.

The final is the sum of the probability of the real person: the real probability of the temporary specific indicator of the current second cycle.

The sum of the final activity scores: the activity score of the current second period temporary specific indicator divided by the update ID, multiplied by the number of IP updates.

In addition, the IP credit smear data may be generated according to the third party IP blacklist, and the IP credit smear data is added to the final specific indicator of the current second period.

Specifically, the information in the blacklist generally includes a blacklisted IP address or an IP address segment. The credit smear data may preferably be expressed in the form of a credit smear score, for example, the more third party blacklists are present, the higher the credit smear score, and the credit smudge score is zero if it does not exist in the blacklist.

The final specific indicator of the current second period is used to cover the final specific indicator of the previous second period, and the number of times to update the final specific indicator of the second period and the number of updates to the corresponding IP address are recorded. Update all of the above data to MySQL.

When the temporary specific indicator and the final specific indicator of the second period are stored in MySQL, the IP can be used as an index; or the IP can be converted to a corresponding long integer and then indexed by the corresponding long integer, and the IPInt is divided into 256 parts and then stored in a sub-table. Easy to query and improve the speed of the query.

After obtaining the final specific indicator updated according to the current second period, the final specific indicator corresponding to the IP address is used as the credit data of the IP address, and the IP address is evaluated according to the credit data. Providing an interface to the third party, allowing access to the credit data of the IP address through the interface; or receiving an IP verification request for the IP address by the third party, searching for credit data corresponding to the IP address, according to the credit data The credit rating is evaluated on the IP address, and the evaluation result is returned to the third party.

It can be applied to the operation of the IP interception by the firewall. The firewall determines the legality of the IP address based on the credit data of the IP address, and can also independently form an IP credit rating platform to provide IP verification results to the firewall. It can also provide a secondary verification mechanism without affecting the existing firewall function. That is, when the firewall determines that the IP address is suspicious, the IP credit rating platform performs secondary verification based on the credit data to further improve the accuracy of the firewall interception. Sex, prevent mistakes.

The IP address analysis method provided by the embodiment of the present invention can be combined with the existing Internet architecture. As shown in FIG. 2, the user access log is collected as the original log, combined with the third party IP blacklist and IP address library, and the present invention is used. The IP address analysis method provided by the embodiment obtains IP user attribute data mainly composed of final specific indicators, IP stain data mainly generated according to a third-party IP blacklist, and IP address pool data obtained according to a third-party IP address library, and IP user attribute data, IP smear data and IP address pool data are integrated into the IP credit rating platform, and the IP address is credit-rated to obtain credit data of the IP address. The credit data of the IP address comprehensively describes the characteristics of the IP address, and can be used for confirming the security of the IP address in the information security field, or realizing the IP address based on the big data analysis. The application results can also be fed back to the IP credit rating platform to perform algorithm iteration and parameter adjustment on the existing results, giving the system the ability to self-learn and self-adjust, and further improve the accuracy of the IP credit rating platform for IP analysis.

Embodiment 2 of the present invention will be described below with reference to the accompanying drawings.

An embodiment of the present invention provides an IP address analysis system, and the structure thereof is as shown in FIG. 3, including:

Big data computing platform and offline computing platform.

Among them, the big data platform includes: Hadoop computing platform, spark computing platform;

Offline computing platforms include: servers or server clusters.

The offline computing platform is further configured to communicate with a third party through the big data platform, provide credit data of the IP address to a third party, or receive a third party query request to return an authentication according to the IP address of the credit data. information.

Preferably, the IP address analysis system further includes a storage platform, the storage platform supports a MySQL system, and can be configured to store the original log, credit data of an IP address, a third-party IP library obtained from a third party, and a third-party IP black. The list, the final specific indicators of the current cycle, the final specific indicators of the previous second cycle, the temporary specific indicators of the current second cycle, and the intermediate data generated during the operation.

The embodiment of the present invention further provides a computer readable storage medium, where the computer program stores a computer program, and when the program is executed by the processor, the steps of the foregoing method are implemented.

The embodiment of the invention further provides a computer device, comprising a memory, a processor and a computer program stored on the memory and operable on the processor, the processor implementing the program to implement the steps of the above method.

The embodiment of the present invention provides an IP address analysis system, which can be combined with an IP address analysis method provided by an embodiment of the present invention to collect historical data of an IP address and analyze historical data of the IP address. The credit data of the IP address realizes accurate analysis of IP address refinement, determines the IP address attribute with big data, and has a comprehensive and accurate understanding of the IP address credit status, which can be applied to IP address legality verification, IP address interception, etc. In the scenario, the problem of IP address security cognition error is solved, and the IP address legality misjudgment and IP address misinterpretation are effectively prevented.

The above description may be implemented individually or in combination in various ways, and such modifications are within the scope of the invention.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or equivalently replace some of the technical features. Modifications or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the invention.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and functional blocks/units of the methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical The components work together. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and nonvolatile, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Sex, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may Any other medium used to store the desired information and that can be accessed by the computer. Moreover, it is well known to those skilled in the art that communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. .

Industrial applicability

In the embodiment of the present invention, by collecting the historical data of the IP address, analyzing the historical data of the IP address, generating the credit data of the IP address, realizing the accurate analysis of the IP address refinement, and determining the IP address attribute by the big data. A comprehensive and accurate understanding of the IP address credit situation can be applied to scenarios such as IP address legality verification and IP address interception, which solves the problem of IP address security cognitive error, effectively preventing IP address legality misjudgment, IP The occurrence of an address error.

Claims

An IP address analysis method, including:

Collect historical data of IP addresses;

The historical data of the IP address is analyzed to generate credit data of the IP address.
The IP address analysis method according to claim 1, wherein the step of collecting historical data of the IP address comprises:

Collect and parse the original log in the first cycle;

Formatting the information in the original log to obtain a predefined indicator, the predefined indicator including at least one or more of the following information:

Time, IP, working time request number, rest period request number, sleep period request number, working period request file size, rest period request file size, sleep period request file size, working period user agent UserAgent number, rest period UserAgent number, sleep Number of time UserAgents, number of mobile UserAgents, number of UserAgents on the PC, number of access sources, number of visited domain names, number of hours of occurrence;

The predefined indicators corresponding to the respective IP addresses are stored, and each IP address corresponds to one or more predefined indicators.
The IP address analysis method according to claim 2, wherein the step of collecting historical data of the IP address further comprises:

In the second cycle, third-party IP libraries and/or third-party IP blacklists are obtained from third-party platforms one or more times.
The IP address analysis method according to claim 3, wherein analyzing the historical data of the IP address and generating the credit data of the IP address comprises:

Pre-defining the pre-defined indicators of the first period corresponding to the working days in the second period and normalizing to obtain intermediate values of the respective working days, and pre-defining the first period corresponding to the rest days in the second period The indicator is pre-processed and normalized to obtain an intermediate value of each rest day, where the second period includes a plurality of first periods corresponding to the working days and a plurality of first periods corresponding to the rest days;

Performing weighted average processing on the intermediate values of the respective working days to obtain a working day weighted average value;

Performing a weighted average or maximum value for each of the rest day intermediate values to obtain a rest day weighted mean or maximum value;

The current second period temporary specific indicator in the second period is calculated according to one or more working day weighted averages and one or more rest day weighted average values, and the current second period temporary specific indicators include:

This week is the probability of office export IP. This period is the probability of household export IP. This period is the probability of real person, the activity score of this cycle, and the number of people in this cycle.

Adjusting the current second periodic temporary specific indicator according to the final specific indicator of the previous second period and the third-party IP library and/or the third-party IP blacklist, and obtaining the final specific indicator of the current second period, The final specific indicator of the second period serves as credit data for the IP address.
The IP address analysis method according to claim 4, wherein the pre-defined and normalized first-period indicators corresponding to the working day in the second period are pre-processed and normalized to obtain intermediate values of the respective working days. The steps of pre-processing and normalizing the predefined indicators of the first period corresponding to the rest day in the second period to obtain the intermediate values of each rest day include:

Calculate the first cycle hour number score corresponding to the working day, the mobile userAgent score, the PC userAgent score, the rest period VS work time request number score, the rest period VS sleep time request number score, the access domain name score, and the above The score is taken as a weighted mean, and the working day is the intermediate value of the household exit IP probability;

Calculating the first cycle working period request number score corresponding to the working day, the working period VS rest period request number score, the rest period VS sleep period request number score, the PC end UserAgent number score, the working period VS rest period UserAgent number score, and the above The weighted average of the points is obtained, and the working day is the intermediate value of the office exit IP probability;

Calculate the first cycle request number distribution score, the UserAgent number distribution score, the hour appearance number score, the domain name number VS source number score, the mobile terminal VS PC end UserAgent number score corresponding to the working day, and the weighted average value of the above scores is obtained, and the working day is obtained. The median probability of being a real person;

Calculating the first period access domain number score, the working period request number score, the rest period request number score, the sleep period request number score, the hour appearance number score, the request source number score corresponding to the working day, and weighting the average score of the above scores, Intermediate value of working day activity;

Calculate the first cycle hour number score corresponding to the rest day, the mobile userAgent score, the PC end UserAgent number score, the rest period VS work period request number score, the rest period VS sleep period request number score, the access domain name score, and the above The score takes the weighted mean value, and the rest day is the intermediate value of the household exit IP probability;

Calculate the first period rest period request number score, the UserAgent number distribution score, the hour appearance number score, the domain name number VS source number score, the mobile end VS PC end UserAgent number score corresponding to the rest day, and take the weighted mean value of the above scores to obtain a rest The daily value of the probability of a real person;

Calculating the first period access domain number score, the work period request number score, the rest period request number score, the sleep period request number score, the hour appearance number score, the request source number score corresponding to the rest day, and weighting the average score of the above scores, The median value of the rest day activity.
The IP address analysis method according to claim 4, wherein the step of calculating the current second period temporary specific indicator in the second period according to one or more working day weighted average values and one or more rest day weighted average values include:

After pre-processing and normalization, the number of UserAgents on the workday VS rest day, the number of workdays on the workday VS rest days, the number of workdays VS rest days, and the number of breaks on the rest days are obtained. The weighted average value of the intermediate value of the office exit IP probability is the probability that the current period is the office exit IP;

The working day is the intermediate value of the household export IP probability and the rest day is the weighted average of the intermediate value of the household export IP probability as the current period is the probability of the household export IP;

The working day is the weighted mean of the median probability of the real person and the rest day as the median probability of the real person as the true probability of the cycle;

The weighted average of the median activity day value and the median value of the rest day activity is used as the cycle activity score.

The number of mobile Agents on the working day and the number of mobile Agents on the rest day and the maximum number of PCs on the workday and the number of UserAgents on the rest of the PC are grouped as the number of people in this cycle.
The IP address analysis method according to claim 4, wherein the final specific indicator includes at least one or more of the following information.

IP, IPInt, update ID, the number of IP updates, the final number of people grouped, and finally the sum of the probability of office export IP, and finally the sum of the probability of household export IP, and finally the sum of the probability of real people, the sum of the final activity scores,

The IPInt is a long integer corresponding to the IP address, the Update ID is the number of times to update the final specific indicator of the second period, and the IP update number is the final specific indicator of the second period of the update of the IP address. frequency,

The step of adjusting the temporary specific indicator of the current second period according to the final specific indicator of the previous second period and the third-party IP library and/or the third-party IP blacklist to obtain the final specific indicator of the current second period include:

For the IP address involved in the final specific indicator of the last second period and the temporary specific indicator of the current second period, the final specific indicator of the current second period is obtained by the following calculation:

When the current number of people of the temporary specific indicator in the second period and the final number of the final specific indicator of the previous second period are grouped into adjacent groups, the group with the large number of people is selected as the final number of groups in the current second period, otherwise the current selection is selected. The number of people in the second cycle of temporary specific indicators,

The sum of the probability of the office exit IP in the temporary specific indicator of the second period plus the probability of the final office outlet IP in the final specific indicator of the second period is the sum of the probability of the final office IP of the current second period.

The sum of the probability of the household export IP plus the probability of the final household IP in the final specific indicator of the second period in the temporary specific indicator of the second period is the sum of the probability of the final household IP of the current second period.

The sum of the probability of the real person in the temporary specific indicator of the second period plus the final probability of the real person in the final specific indicator of the second period is the sum of the probability of the real person in the current second period,

After the sum of the activity score in the temporary specific indicator of the second period plus the final activity score of the second period, divided by the number of final updates recorded by the update ID, multiplied by the number of IP updates, as the current second period The sum of the final activity scores;

For the IP address involved in the temporary specific indicator of the current second period, which is not involved in the final specific indicator of the last second period, obtain the temporary specific indicator of the current second period by the following calculation:

Grouping the number of people in the temporary specific indicator of the current second period as the final number of people in the current second period.

Taking the IP of the office exit IP in the temporary specific indicator of the current second period as the sum of the probability of the final office IP of the current second period,

Taking the probability of household export IP in the temporary specific indicator of the current second period as the sum of the probability of household export IP in the current second period,

Taking the probability of being a real person in the temporary specific indicator of the current second period as the sum of the final real probability of the current second period,

Dividing the activity score in the temporary specific indicator of the current second period by the update ID, multiplying the IP update count as the sum of the final activity scores of the current second period;

The final specific indicator of the current second period is used to cover the final specific indicator of the previous second period, and the number of times to update the final specific indicator of the second period and the number of updates to the corresponding IP address are recorded.
The IP address analysis method according to claim 7, wherein the current specific indicator of the current second period is based on the final specific indicator of the last second period and the third-party IP library and/or the third-party IP blacklist. The steps of adjusting to obtain the final specific indicator of the current second cycle include:

Filtering data corresponding to the IP non-syntax or the corresponding IP as the local area network IP in the temporary specific indicator of the current second period;

According to the additional information of the IP address included in the third-party IP library, adjust the IP probability of the office exit, the probability of the household exit IP, and the probability of the real person in the temporary specific indicators of the current second period;

Generating IP credit smear data according to the third-party IP blacklist, and adding the IP credit smear data to the final specific indicator of the current second period.
The IP address analysis method according to claim 1, wherein the method further comprises:

Providing an interface to a third party to allow access to credit data of the IP address through the interface; or

Receiving a third party to issue an IP verification request for the IP address, searching for credit data corresponding to the IP address, performing credit rating evaluation on the IP address according to the credit data, and returning the evaluation result to the third party.
An IP address analysis system, including a big data platform and an offline computing platform;

The big data platform is configured to store original logs, calculate original logs, and collect and store historical data of IP addresses;

The offline computing platform is configured to analyze historical data of an IP address collected by the big data platform to generate credit data of an IP address.
A computer readable storage medium having stored thereon a computer program, the program being executed by a processor to perform the steps of the method of any one of claims 1 to 9.
A computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, the processor executing the program to implement any one of claims 1 to The steps of the method.