CN108923967B

CN108923967B - Duplication-removing flow recording method, duplication-removing flow recording device, server and storage medium

Info

Publication number: CN108923967B
Application number: CN201810686101.7A
Authority: CN
Inventors: 赵吉
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2018-06-27
Filing date: 2018-06-27
Publication date: 2021-07-02
Anticipated expiration: 2038-06-27
Also published as: CN108923967A; WO2020000742A1

Abstract

The embodiment of the invention discloses a duplicate removal flow recording method, a duplicate removal flow recording device, a server and a storage medium, wherein the method comprises the following steps: under the condition of receiving the message, acquiring field point information of the message, determining whether the flow corresponding to the message is duplication-removing flow matched with preset flow configuration information or not according to the field point information, if so, analyzing the field point information, determining service generation time and indication information of the duplication-removing flow, determining a plurality of service time windows for recording the duplication-removing flow according to the preset flow configuration information and the service generation time, and further respectively recording the duplication-removing flow under the plurality of service time windows according to the indication information. According to the invention, the de-weight flow is recorded in a time window mode, so that the de-weight flow can be recorded more accurately.

Description

Duplication-removing flow recording method, duplication-removing flow recording device, server and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a duplicate removal traffic recording method, apparatus, server, and storage medium.

Background

With the development and popularization of internet technology, the number of users corresponding to each platform is greatly increased, and the data traffic corresponding to the users is also increased in geometric number, which forms a huge challenge for data statistics and analysis work. In many statistical traffic data, the index of the deduplication count is a basic and common statistical index, and the determination of the deduplication count depends on statistics of the deduplication traffic. Therefore, how to establish better deduplication logic to record deduplication traffic becomes a problem to be solved urgently.

Disclosure of Invention

The embodiment of the invention provides a method, a device, a server and a storage medium for recording de-weight flow, which are used for recording the de-weight flow in a time window mode and are beneficial to more accurately recording the de-weight flow.

In a first aspect, an embodiment of the present invention provides a deduplication traffic recording method, where the method includes:

under the condition of receiving a message, acquiring field point information of the message, and determining whether the flow corresponding to the message is a duplicate removal flow matched with preset flow configuration information or not according to the field point information;

if the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information, analyzing the field point information and determining the service generation time and the indication information of the duplication elimination flow, wherein the indication information is used for indicating to execute flow adding operation or flow subtracting operation on the duplication elimination flow;

determining a plurality of service time windows for recording the deduplication traffic according to the preset traffic configuration information and the service generation time, wherein each service time window in the plurality of service time windows corresponds to different time granularities;

and respectively recording the duplication elimination flow under the plurality of service time windows according to the indication information.

In an embodiment, the preset traffic configuration information includes a recorded time precision, and the specific implementation manner of determining the plurality of service time windows for recording the deduplication traffic according to the preset traffic configuration information and the service generation time is as follows:

acquiring the recorded time precision from the preset flow configuration information, and determining a minimum time granularity window and a sub-small time granularity window according to the recorded time precision, wherein the time granularity of the minimum time granularity window is smaller than that of the sub-small time granularity window;

determining the minimum time granularity window corresponding to the service generation time as a minimum service time window, and determining the sub-small time granularity window corresponding to the service generation time as a sub-small service time window;

determining a plurality of business time windows for recording the deduplication traffic based on the minimum business time window and the second minimum business time window.

In an embodiment, the specific implementation manner that the indication information is used to indicate that a traffic adding operation is performed on the deduplication traffic, and according to the indication information, the deduplication traffic is recorded under the multiple service time windows respectively is as follows:

analyzing the indication information, and judging whether the duplication-removing flow is a repeated flow under the condition that the indication information is used for indicating to execute the flow adding operation on the duplication-removing flow;

if the de-duplication traffic is not the repeated traffic, performing an increasing operation on the occurrence frequency of the de-duplication traffic under the plurality of service time windows respectively;

and if the deduplication traffic is the repetition traffic, keeping the occurrence frequency of the deduplication traffic under each service time window in the plurality of service time windows unchanged.

In an embodiment, the indication information is used to indicate that a traffic reduction operation is performed on the deduplication traffic, and a specific implementation manner of recording the deduplication traffic under the multiple service time windows respectively according to the indication information is as follows: analyzing the indication information, and determining whether the indication information is used for indicating to execute the flow reduction operation on the de-duplicated flow; and if the indication information is used for indicating that the flow reduction operation is performed on the de-duplication flow, performing the reduction operation on the occurrence frequency of the de-duplication flow under the plurality of service time windows respectively.

In an embodiment, the preset flow configuration information further includes a time length to failure threshold, and after the deduplication flows are respectively recorded under the multiple service time windows, a timing function may be started for the multiple service time windows; and deleting the recorded data of the deduplication flow under the multiple time windows under the condition that the current timed length is greater than or equal to the failure duration threshold.

In an embodiment, the multiple service time windows include the minimum service time window and the second minimum service time window, and the specific implementation manner of determining whether the deduplication traffic is the duplicate traffic is as follows: determining the current times of recording of the duplication elimination flow under the minimum service time window; and under the condition that the current times is greater than or equal to a time threshold value, judging the deduplication flow as a repeated flow.

In an embodiment, the preset traffic configuration information further includes traffic rule information, the traffic rule information includes a statistical duration, and after determining whether the traffic corresponding to the packet is a duplicate removal traffic matched with the preset traffic configuration information according to the field point information, the method may further include: acquiring the current system time and the statistical duration in the flow rule information; determining a statistical time period for the de-emphasis flow according to the current system time and the statistical duration; acquiring the recorded data of all the service time windows in the statistical time period; and if the duplicate removal flow is determined to meet the flow rule corresponding to the flow rule information according to the recorded data, alarm information is output.

In a second aspect, an embodiment of the present invention provides a deduplication traffic recording apparatus, which includes modules for executing the method of the first aspect.

In a third aspect, an embodiment of the present invention provides a server, which includes a processor, a network interface, and a memory, where the processor, the network interface, and the memory are connected to each other, where the network interface is controlled by the processor to send and receive messages, the memory is used to store a computer program that supports the server to execute the above method, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect.

In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, the computer program comprising program instructions that, when executed by a processor, cause the processor to perform the method of the first aspect.

In the embodiment of the invention, under the condition that a server receives a message, the field point information of the message is obtained, whether the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information is determined according to the field point information, if yes, the field point information is analyzed, the service generation time and the indication information of the duplication elimination flow are determined, a plurality of service time windows for recording the duplication elimination flow are determined according to the preset flow configuration information and the service generation time, and then the duplication elimination flow is respectively recorded under the plurality of service time windows according to the indication information. The time window form is adopted to record the de-weight flow, which is beneficial to more accurately recording the de-weight flow.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic flow chart of a deduplication traffic recording method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an operation interface for flow configuration according to an embodiment of the present invention;

fig. 3 is a schematic flow chart of another deduplication traffic recording method provided by the embodiment of the present invention;

fig. 4 is a schematic diagram of an operation interface for configuring a flow rule according to an embodiment of the present invention;

FIG. 5 is a schematic block diagram of a deduplication traffic recording apparatus provided by an embodiment of the present invention;

fig. 6 is a schematic block diagram of a server according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a schematic flow chart of a deduplication traffic recording method according to an embodiment of the present invention, and as shown in the drawing, the deduplication traffic recording method may include:

101. the method comprises the steps that under the condition that a message is received by a server, field point information of the message is obtained, and whether the flow corresponding to the message is the duplicate removal flow matched with preset flow configuration information or not is determined according to the field point information.

In one embodiment, a user may configure traffic in advance through a traffic configuration page, where the purpose of the configuration is to determine what kind of traffic is to be recorded. The duplication elimination traffic is traffic concerning different duplication elimination values, and the duplication elimination value may be a unique identifier, such as a device number, an ID number, an account number, and the like. For example, a flow is the number of transactions of a user with the same device number but different device numbers, where the unique identifier of the user may be a user account, and then the user account is a deduplication value, and the flow focuses on different users (i.e., deduplication values), and then is a deduplication flow.

Further, the specific content of the duplicate traffic removal configuration (i.e., the preset traffic configuration information) may include time precision (e.g., minutes) of the record, traffic description information (i.e., description of traffic, such as that one piece of traffic may be described as the number of successful transactions of users with the same device number and different user numbers), a statistical dimension (e.g., an IP address), a statistical field (which may correspond to different user account numbers, for example), a time type, and an identifier corresponding to an adding operation or a subtracting operation performed during the traffic recording, where the identifier corresponding to the adding operation is "001" and the identifier corresponding to the subtracting operation is "002". When the user completes the configuration of the duplication-removing traffic, the server can acquire the traffic configuration information when needed, that is, the user configures the specific content through the traffic configuration page.

The flow configuration page may be as shown in fig. 2, and as can be seen from fig. 2, the deduplication flow (i.e., flow description information) of "transaction times of users with the same device number" is configured, the statistical dimension is, for example, an IP address, the statistical field is an account unique identifier (e.g., a device number), the time precision of the record is minutes, the addition operation corresponding identifier is "001" and the subtraction operation corresponding identifier is "002" during flow recording.

In an embodiment, when the server receives the message, the field point information of the message may be compared with the traffic configuration information in step 101, and then it is determined whether the traffic corresponding to the message is the deduplication traffic matched with the preset traffic configuration information according to the comparison result. The message may include a plurality of field points, and each field point in the plurality of field points and the value corresponding to each field point are the field point information of the mentioned message.

For example, assuming that the traffic configuration page is as shown in fig. 2, the configured deduplication traffic is the number of successful transactions of users with the same device number and different device numbers, and the messages received by the system are as follows:

the "event timestamp", "event id", "user name", "ip", and "device" are all field points included in the packet, and values corresponding to the field points respectively represent the service generation time "2018-05-0316: 11: 34 ", a transaction success status code" 001 "corresponding to the message, a user name, an ip address" 1.1.1.1 ", a device number, and the like. The field point 'event timestamp' corresponds to the time type in the flow configuration page; the value of the field point 'device' corresponds to the statistical dimension in the flow configuration page; the field point "user name" corresponds to a statistical field in the traffic configuration page, that is, different accounts (such as users U1, U2, or U3) are characterized; the value of "event id" corresponds to an identifier in the traffic configuration page, "event id 001" corresponds to an identifier "001" corresponding to an operation in the traffic configuration page, and "event id 002" corresponds to an identifier "002" corresponding to an operation in the traffic configuration page.

In this case, after receiving the message, the server may obtain, by parsing the message, the event timing, the device name, the user name, and the like of each field point corresponding to the message, and the value corresponding to each field point, compare the obtained field points and the corresponding values with the time type, the statistical dimension, and the statistical field in the traffic configuration information, may determine that each field point is matched with the traffic configuration information, and may determine that the traffic corresponding to the message is the deduplication traffic matched with the traffic configuration information.

102. And if the server determines that the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information, analyzing the field point information and determining the service generation time of the duplication elimination flow and indicating information, wherein the indicating information is used for indicating to execute flow adding operation or flow subtracting operation on the duplication elimination flow.

In an embodiment, after determining that a traffic corresponding to the packet is a deduplication traffic, on one hand, a service generation time of the deduplication traffic may be determined according to a value corresponding to an event time field point (e.g., "event timestamp") in the packet, for example, a value corresponding to "event timestamp" is 2018-05-0316: 11: 34, the service generation time can be determined to be 11 minutes and 34 seconds at 16 points on 3 months in 2018. On the other hand, by identifying the value (e.g., "001" or "002") corresponding to the field point (e.g., "device") of the transaction code, it is determined whether to perform the traffic addition operation or the traffic subtraction operation on the piece of deduplication traffic, that is, to determine the indication information for the piece of deduplication traffic. For example, when the server recognizes that the value corresponding to the field point "device" is "001", the server may generate instruction information for instructing to perform a traffic addition operation on the duplication-removed traffic, and when the value corresponding to the field point "device" is "002", the server may generate instruction information for instructing to perform a traffic subtraction operation on the duplication-removed traffic.

103. The server determines a plurality of service time windows for recording the deduplication traffic according to preset traffic configuration information and service generation time, wherein each service time window in the plurality of service time windows corresponds to different time granularities.

In one embodiment, the preset traffic configuration information includes recorded time precision, and the server may obtain the recorded time precision from the preset traffic configuration information, determine a minimum time granularity window and a sub-minimum time granularity window according to the recorded time precision, further determine the minimum time granularity window corresponding to the service generation time as a minimum service time window, determine the sub-minimum time granularity window corresponding to the service generation time as a sub-minimum service time window, and determine a plurality of service time windows for recording the deduplication traffic based on the minimum service time window and the sub-minimum service time window. Wherein the time granularity of the minimum time granularity window is less than the time granularity of the next time granularity window.

The time granularity of the minimum time granularity window may be consistent with the time granularity corresponding to the time precision of the record, that is, the time precision of the record is minutes, and then the time granularity of the minimum time granularity window may also be minutes.

Illustratively, the time precision recorded in the traffic configuration information is minutes, and the service generation time of the deduplication traffic is 16: 11: 23 seconds of No. 5/3 in 2018. In this case, the server may determine the minimum time granularity window as a minute-level window according to the time accuracy, and the next-hour granularity window may be determined as an hour-level window and a day-level window. Further, the server may combine the traffic generation time of the deduplication traffic, the number of 2018-05-03T 16: 11: 00 is determined as a minute level window corresponding to the service generation time (i.e. 16: 11: 23 seconds of No. 5/3 in 2018), namely determined as a minimum service time window; mixing 2018-05-03T 16: 00: 00 and 2018-05-03T 00: 00: 00 are respectively determined as an hour level window and a day level window corresponding to 16 points, 11 minutes and 23 seconds of No. 5 and No. 3 in 2018, namely determined as a second-time small business time window. Further, based on the determined minimum traffic time window and the next minimum traffic time window, the time windows 2018-05-03T16 may be: 11: 00. 2018-05-03T 16: 00: 00 and 2018-05-03T 00: 00: 00 are determined as a plurality of traffic time windows for recording deduplication traffic. It can be seen that each business time window is prioritized according to time granularity, where the minute level is the minimum level, the hour level is the next, and the day level is the next.

104. And the server records the duplication elimination flow under a plurality of service time windows respectively according to the indication information.

The indication information is used for indicating that the flow adding operation or the flow subtracting operation is carried out on the de-duplicated flow. In one embodiment, when the server recognizes that the indication information is used to indicate that a traffic adding operation is performed on the deduplication traffic, it may first determine whether the deduplication traffic is a duplicate traffic, and if the deduplication traffic is the duplicate traffic, not record the deduplication traffic; and if the traffic is not the repeated traffic, performing increasing operation on the occurrence number of the de-duplicated traffic under each determined service time window.

Or, when the server recognizes that the indication information is used to indicate that the traffic reduction operation is performed on the deduplication traffic, the reduction operation may be performed on the number of occurrences of the deduplication traffic in each determined service time window.

In the embodiment of the invention, under the condition that a server receives a message, the field point information of the message is obtained, whether the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information is determined according to the field point information, if yes, the field point information is analyzed to determine the service generation time and the indication information of the duplication elimination flow, a plurality of service time windows for recording the duplication elimination flow are determined according to the preset flow configuration information and the service generation time, and then the duplication elimination flow is respectively recorded under the plurality of service time windows according to the indication information. According to the embodiment of the invention, the deduplication flow is recorded in a time window mode, so that the deduplication flow can be recorded more accurately.

Referring to fig. 3, fig. 3 is a schematic flow chart of another deduplication traffic recording method according to an embodiment of the present invention, where as shown in the drawing, the deduplication traffic recording method may include:

301. the method comprises the steps that under the condition that a message is received by a server, field point information of the message is obtained, and whether the flow corresponding to the message is the duplicate removal flow matched with preset flow configuration information or not is determined according to the field point information.

302. And if the server determines that the flow corresponding to the message is the duplicate removal flow matched with the preset flow configuration information, analyzing the field point information and determining the service generation time and the indication information of the duplicate removal flow.

303. And the server determines a plurality of service time windows for recording the deduplication flow according to the preset flow configuration information and the service generation time.

For specific implementation manners of steps 301 to 303, reference may be made to the related descriptions of steps 101 to 103 in the foregoing embodiments, and details of the present invention are not described herein again.

304. And the server records the duplication elimination flow under a plurality of service time windows respectively according to the indication information.

In an embodiment, the server may analyze the indication information, determine whether the deduplication traffic is a duplicate traffic when it is determined that the indication information is used to indicate that a traffic adding operation is performed on the deduplication traffic, perform an adding operation on the number of occurrences of the deduplication traffic in a plurality of service time windows if the deduplication traffic is not the duplicate traffic, and keep the number of occurrences of the deduplication traffic in each of the plurality of service time windows unchanged if the deduplication traffic is the duplicate traffic.

The specific implementation manner of the server determining whether the duplicate removal traffic is the duplicate traffic may be: and the server determines the current times of the deduplication flow recorded under the minimum service time window, and judges the deduplication flow as the repetitive flow under the condition that the current times is greater than or equal to a time threshold.

For example, assume that a deduplication flow is described as follows: the transaction success times of the users with the same equipment number are different. Each transaction is successful by default, so by default the value is incremented by 1 at the point in time when the transaction occurred. For example, the received transaction message is a, and a transaction message B is received before the transaction message. The transaction message a represents that the 16 o 'clock, 11 o' clock, 54 s user U1, No. 5/3 in 2018, transacts through the device D, and when a transaction request is initiated, the system defaults that the transaction is successful (most of the cases are also successful) when the message a is received. The transaction message B represents that a user U1 transacts at 16 o' clock, 11 min, 23 sec, No. 5/3 in 2018 through a device D (which may be a mobile phone, a computer, a tablet, or the like), when a transaction request is initiated, the system defaults that the transaction is successful (most of cases are successful) when the message B is received, and records of the deduplication traffic corresponding to the message B under each service time window are as follows:

device D-2018-05-03T 16: 11: 00 → [ (U1, 1) ], for characterizing device D in 2018-05-03T 16: 11: a time point of 00 (time window in minutes) was recorded as [ (U1, 1) ];

device D-2018-05-03T 16: 00: 00 → [ (U1, 1), for characterizing device D in 2018-05-03T 16: 00: a time point of 00 (hour scale time window) has a record of [ (U1, 1) ];

device D-2018-05-03T 00: 00: 00 → [ (U1, 1) ], for characterizing device D in 2018-05-03T 00: 00: the time point of 00 (time window on the order of days) was recorded as [ (U1, 1) ].

In this case, when the transaction message a is received, the message a is analyzed, and it is determined that the deduplication value corresponding to the message a is U1, that is, the user is U1; the corresponding service generation time is 16: 11: 23 seconds of No. 5/3 in 2018, and the minimum time window corresponding to the service generation time, namely the minimum service time window is 2018-05-03T 16: 11: 00, record for deduplication traffic of U1 under this minimum traffic time window [ (U1, 1) ], by analyzing the minimum traffic time window 2018-05-03T 16: 11: a value of 00, it can be determined that the deduplication flow for U1 is between 2018-05-03T 16: 11: 00, that is, the number of times U1 is greater than or equal to 1 (that is, the number threshold), it can be determined that the deduplication traffic is the duplicate traffic, and the deduplication traffic number of all the service time windows U1 of 16 o 11 min 23 s, No. 5/3 in 2018 is kept unchanged, that is, the records of each service time window are still:

device D-2018-05-03T 16: 11: 00 → [ (U1, 1) ]

Device D-2018-05-03T 16: 00: 00 → [ (U1, 1) ]

Device D-2018-05-03T 00: 00: 00 → [ (U1, 1) ]

Further, time is shifted backwards, a transaction message C is received at 16 o 'clock 11 min 55 sec at 5/3/2018, and the transaction message C represents that a user U2 transacts through the device D at 16 o' clock 11 min 54 sec at 5/3/2018. Analyzing the message C, determining that the duplication elimination value corresponding to the message C is U2, the service generation time corresponding to the message C is 16: 11: 55 seconds of No. 5/3 in 2018, the minimum service time window corresponding to the service generation time is (2018-05-03T 16: 11: 00), and the data recorded under the minimum service time window is [ (U1, 1) ], and analyzing the data in 2018-05-03T 16: 11: 00, it can be determined that the deduplication value U2 corresponds to the deduplication flow rate of U1 in the range of 2018-05-03T 16: 11: 00, if it does not appear in the window, it may be determined that the deduplication traffic is not the duplicate traffic, and then a traffic increasing operation on the deduplication traffic of U2 is performed in all the traffic time windows corresponding to 16 o' clock 11 min 55 sec of No. 5/3 in 2018, and the result after performing the increasing operation is as follows:

device D-2018-05-03T 16: 11: 00 → [ (U1, 1), (U2, 1) ]

Device D-2018-05-03T 16: 00: 00 → [ (U1, 1), (U2, 1) ]

Device D-2018-05-03T 00: 00: 00 → [ (U1, 1), (U2, 1) ]

In an embodiment, the server may further parse the indication information, determine whether the indication information is used to indicate that the traffic reduction operation is performed on the duplicate traffic, and if the indication information is used to indicate that the traffic reduction operation is performed on the duplicate traffic, perform the reduction operation on the number of times that the duplicate traffic occurs under the multiple service time windows, respectively.

Illustratively, the transaction messages A, B and C are received in the previous example, and the time is shifted backwards on the basis of recording the duplicate removal flow rates corresponding to A, B and C in each service time window, and a transaction message F is received after 16 o 'clock 14 min 43 sec of No. 5/3 in 2018, and represents that the transaction of the user U1 fails through the device D at 16 o' clock 11 min 23 sec of No. 5/3 in 2018. And analyzing the message F, determining that the duplication elimination value corresponding to the message F is U1, the corresponding service generation time is 2018, 5, month and 3, 16 points, 11 minutes and 23 seconds, and the indication information is used for performing flow rate reduction operation on the flow duplication elimination. In this case, all current service time windows and their respective corresponding values are obtained as follows:

device D-2018-05-03T 16: 11: 00 → [ (U1, 1), (U2, 1) ]

Device D-2018-05-03T 16: 00: 00 → [ (U1, 1), (U2, 1) ]

Device D-2018-05-03T 00: 00: 00 → [ (U1, 1), (U2, 1) ]

Then the result of performing a traffic subtraction operation on the de-duplicated traffic of U1 under all traffic time windows is as follows:

device D-2018-05-03T 16: 11: 00 → [ (U1, 0), (U2, 1) ]

Device D-2018-05-03T 16: 00: 00 → [ (U1, 0), (U2, 1) ]

Device D-2018-05-03T 00: 00: 00 → [ (U1, 0), (U2, 1) ]

In an embodiment, the preset traffic configuration information further includes a time length of failure threshold, and after the server records the duplicate removal traffic in the multiple service time windows, the server may further start a timing function for the multiple service time windows, and delete the recorded data of the duplicate removal traffic in the multiple time windows when the current timed time length is greater than or equal to the time length of failure threshold.

For example, the time length threshold of the failure is 1 hour, and after the server records the deduplication traffic under a plurality of service time windows respectively, the recording data under the plurality of service time windows are:

device D-2018-05-03T 16: 11: 00 → [ (U1, 1) ]

Device D-2018-05-03T 16: 00: 00 → [ (U1, 1) ]

Device D-2018-05-03T 00: 00: 00 → [ (U1, 1) ]

In this case, after the server records the deduplication traffic in the plurality of service time windows, the server may start a timing function for the plurality of service time windows to record the current start time. For example, the current start time is 2018-05-03T 16: 11: 00, then at the current time is determined by 2018-05-03T 16: 11: 00 to 2018-05-03T 17: 11: when 00 hours exist, the current counted time length can be determined to be equal to 1 hour, and the recorded data under the three service time windows are deleted. By adopting the mode, the memory occupation of the storage space can be reduced.

305. And the server acquires the current system time and the statistical duration in the preset flow rule information.

306. And the server determines a statistical time period for the deduplication flow according to the current system time and the statistical duration.

In one embodiment, a user may configure the traffic rule through a traffic rule page, and the specific configuration content (i.e., the preset traffic rule information) may include: the statistical duration, the traffic rule (if the number of successful transactions of the user is greater than 4, as with the device number), and the configuration page may be as shown in fig. 4. As can be seen from fig. 4, the configured flow rule is that the number of successful transactions of users with the same equipment number is greater than 4, the statistical duration is 1 hour, and the configured flow for executing the flow gauge is the deduplication flow of "number of successful transactions of users with the same equipment number is different".

Further, after the server determines that the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information according to the field point information, whether a duplication elimination flow statistical function is started or not can be detected, if the duplication elimination flow statistical function is started, the current system time is obtained, and a statistical time period for the duplication elimination flow is determined according to the current system time and the statistical duration. In one embodiment, the statistical duration may be divided into an hour level and a minute level, and the statistical time period determined by the server according to the current system time and the statistical duration is different for different time levels. Specifically, when the statistical duration is in the level of hour, one hour represents that an hour window where the time point is located is taken from the time point corresponding to the current system time, for example, the time point corresponding to the current system time is 2018-05-03T 16: 11: 00, then this one hour is 2018-05-03T 16: 00: 00 to 2018-05-03T 16: 11: 00 these 11 minutes, these 11 minutes are the determined statistical time period. Or, when the statistical duration is in the order of minutes, m minutes represents that m (m is a number greater than 0) minutes are pushed back from the time point corresponding to the current system time, for example, m is 60, and the time point corresponding to the current system time is 2018-05-03T 16: 11: 00, then from 2018-05-03T 16: 11: time range 2018-05-03T15 for 60 minutes after 00 starts to push back: 11: 00 to 2018-05-03T 16: 11: 00 this 60 minutes is the determined statistical time period.

307. And the server acquires the recorded data of all the service time windows in the statistical time period.

308. And if the server determines that the duplication removing flow meets the flow rule corresponding to the flow rule information according to the recorded data, outputting alarm information.

Illustratively, the preconfigured traffic rule is that the number of successful transactions of users with the same device number is greater than 10, and the statistical duration determined in step 206 is 2018-05-03T 16: 00: 00 to 2018-05-03T 16: 11: 00 these 11 minutes, the recorded data (i.e., recorded deduplication traffic) for all business time windows in these 11 minutes is as follows:

1) device D-2018-05-04T 10: 08: 00 → [ ] [ ] indicates that there is no space, no flow

2) Device D-2018-05-04T 10: 07: 00 → [ ]

3) Device D-2018-05-04T 10: 06: 00 → [ ]

4) Device D-2018-05-04T 10: 05: 00 → [ ]

5) Device D-2018-05-04T 10: 04: 00 → [ ]

6) Device D-2018-05-04T 10: 03: 00 → [ ]

7) Device D-2018-05-04T 10: 03: 00 → [ ]

8) Device D-2018-05-04T 10: 02: 00 → [ (U101, 1) ]

9) Device D-2018-05-04T 10: 01: 00 → [ (U1, 1), (U2, 1) … … (U10, 1) ] 10 users were recorded

10) Device D-2018-05-04T 10: 00: 00 → [ (U1, 1), (U2, 1) … … (U11, 1) ] 11 users were recorded

11) Device D-2018-05-04T 9: 59: 00 → [ (U1, 1), (U2, 1) … … (U100, 1) ] 100 users are recorded.

Wherein U1-U101 represent different users, i.e. deduplication values. After the server acquires the recorded data of all the service time windows, the data under all the service time windows can be taken out, 101 different users are in total, namely the number of successful transactions of the users with the same equipment number is 101, and the 101 is greater than 10, the duplication removing flow can be judged to meet the flow rule, alarm information can be output, and risk control is facilitated.

In one embodiment, the server may resolve the statistical time period into a time window set composed of a plurality of time windows of different time granularities when counting the deduplication traffic within the statistical time period. For example, the statistical time period is from 11/month 1/2014 to 2016/month 1/2/2014, and the statistical time period can be divided into 5 time periods, i.e., 11/2014, 12/2014, 2015, 2016/month 1/2016, and 2016/month 1/2/2016, according to the start time and the end time of the time period.

For example, assuming that the recorded time precision is minutes, and the statistical duration configured in the flow measurement rule configuration page is 60 × 24 × 2 minutes, the server determines that the time when the flow corresponding to the packet is the deduplication flow matched with the preset flow configuration information is 2018-05-03T 16: 11: 34, since the time precision used when the flow gauge is configured is minutes, the time of 34 seconds is assigned to the time point of 11 minutes, 2018-05-03T 16: 11: 34 should be 2018-05-01T 16: 11: 00, the server can determine that the statistical time period of the deduplication traffic is 2018-05-03T16 according to the statistical duration: 11: 34 back-push 60 x 24 x 2 minutes. When calculating the deduplication flows in the to-be-counted time period, all the time windows at the level of all the minutes in the to-be-counted time period may be taken out, from 2018-05-01T 16: 11: 00 begin, get

2018-05-01T16：12：00

2018-05-01T16：13：00

2018-05-01T16：14：00

…

2018-05-01T16：58：00

2018-05-01T16：59：00

2018-05-01T17：00：00

2018-05-01T17：01：00

2018-05-01T17：02：00

…

2018-05-01T18：00：00

2018-05-01T18：01：00

2018-05-01T18：02：00

…

2018-05-01T19：00：00

2018-05-01T19：01：00

2018-05-01T19：02：00

…

2018-05-01T19：00：00

2018-05-01T19：01：00

2018-05-01T19：02：00

…

2018-05-01T23：58：00

2018-05-01T23：59：00

2018-05-02T00：00：00

2018-05-02T00：01：00

…

2018-05-02T23：58：00

2018-05-02T23：59：00

2018-05-03T00：00：00

2018-05-03T00：01：00

…

2018-05-03T01：00：00

…

2018-05-03T02：00：00

…

2018-05-03T15：59：00

2018-05-03T16：00：00

2018-05-03T16：01：00

…

2018-05-03T16：11：00

A total of 60 x 24 x 2 2880 minute-scale time windows. For this case, it can be identified whether the time windows of the respective minute levels are all in other higher level time dimensions, and if so, a combination of the respective hour and day dimensions can be made. For example, in the above list of minutes, 2018-05-02T 00: 00: 00 to 2018-05-02T 23: 59: 00 is in the time window corresponding to the day 2018-05-02, 2018-05-01T 17: 00: 00 to 2018-05-01T 17: 59: 00 are all within the corresponding hour time window. The combined statistical time period includes a 23-hour window, as shown below:

2018-05-01T17

2018-05-01T18

2018-05-01T19

2018-05-01T20

2018-05-01T21

2018-05-01T22

2018-05-01T23

2018-05-03T00

2018-05-03T01

2018-05-03T02

…

2018-05-03T15

and also comprises 1 day time window corresponding to the day of 2018-05-02 after combination and a minute time window which cannot be combined: 2018-05-01T 16: 12: 00 to 2018-05-01T 16: 59: 00. 2018-05-03T 16: 00: 00 to 2018-05-03T 16: 11: 00, so that the total number of day time windows 1 + hour time window 23 + minute time windows 60 to 84 time windows is required, which is reduced to about 1/40. When the time span corresponding to the flow rule is larger, the optimization effect is more obvious. The number of extracted time windows can be reduced and the statistical efficiency of the deduplication traffic is improved.

In the embodiment of the invention, the server can acquire the field point information of the message under the condition of receiving the message, determine whether the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information or not according to the field point information, if so, analyze the field point information, determine the service generation time and the indication information of the duplication elimination flow, determine a plurality of service time windows for recording the duplication elimination flow according to the preset flow configuration information and the service generation time, and further record the duplication elimination flow under the plurality of service time windows according to the indication information. Further, the server can also obtain the current system time and the statistical duration in the preset flow rule information, determine the statistical time period for the deduplication flow according to the current system time and the statistical duration, obtain the recorded data of all service time windows in the statistical time period, and output alarm information if the server determines that the deduplication flow meets the flow rule corresponding to the flow rule information according to the recorded data. On the one hand, the time window form can be adopted to record the de-weight flow, so that the de-weight flow can be recorded more accurately, and on the other hand, the alarm information can be output under the condition that the de-weight flow meets the flow rule, so that the risk control is facilitated.

The embodiment of the present invention further provides a duplicate removal flow recording apparatus, which is used for executing the modules of the method described in the foregoing fig. 1 or fig. 3. Specifically, referring to fig. 5, a schematic block diagram of a deduplication flow recording apparatus according to an embodiment of the present invention is provided. The duplication removal flow recording apparatus of the present embodiment includes: an acquisition module 50, a determination module 51 and a recording module 52.

An obtaining module 50, configured to obtain field point information of a message when the message is received;

a determining module 51, configured to determine, according to the field point information obtained by the obtaining module 50, whether a traffic corresponding to the packet is a deduplication traffic matched with preset traffic configuration information;

the determining module 51 is further configured to, if it is determined that the traffic corresponding to the packet is a duplicate removal traffic matched with preset traffic configuration information, analyze the field point information and determine service generation time of the duplicate removal traffic and indication information, where the indication information is used to indicate that a traffic adding operation or a traffic subtracting operation is performed on the duplicate removal traffic;

the determining module 51 is further configured to determine, according to the preset traffic configuration information and the service generation time, a plurality of service time windows for recording the deduplication traffic, where each of the plurality of service time windows corresponds to a different time granularity;

a recording module 52, configured to record the deduplication traffic under the multiple service time windows respectively according to the indication information.

In an embodiment, the preset traffic configuration information includes a recorded time precision, and the determining module 51 is specifically configured to obtain the recorded time precision from the preset traffic configuration information, and determine a minimum time granularity window and a sub-small time granularity window according to the recorded time precision, where a time granularity of the minimum time granularity window is smaller than a time granularity of the sub-small time granularity window; determining the minimum time granularity window corresponding to the service generation time as a minimum service time window, and determining the sub-small time granularity window corresponding to the service generation time as a sub-small service time window; determining a plurality of business time windows for recording the deduplication traffic based on the minimum business time window and the second minimum business time window.

In an embodiment, the indication information is used to indicate that a traffic adding operation is performed on the duplication-removing traffic, and the recording module 52 is specifically configured to analyze the indication information according to the indication information, and determine whether the duplication-removing traffic is a duplicate traffic when the indication information is used to indicate that the traffic adding operation is performed on the duplication-removing traffic; if the de-duplication traffic is not the repeated traffic, performing an increasing operation on the occurrence frequency of the de-duplication traffic under the plurality of service time windows respectively; and if the deduplication traffic is the repetition traffic, keeping the occurrence frequency of the deduplication traffic under each service time window in the plurality of service time windows unchanged.

In an embodiment, the indication information is used to indicate that a traffic reduction operation is performed on the duplication-removing traffic, and the recording module 52 is further configured to, according to the indication information, specifically, analyze the indication information and determine whether the indication information is used to indicate that the traffic reduction operation is performed on the duplication-removing traffic; and if the indication information is used for indicating that the flow reduction operation is performed on the de-duplication flow, performing the reduction operation on the occurrence frequency of the de-duplication flow under the plurality of service time windows respectively.

In one embodiment, the preset flow configuration information further includes a time length to failure threshold, and the apparatus further includes: a timing module 53 and a deletion module 54, wherein:

a timing module 53, configured to start a timing function for the plurality of service time windows;

a deleting module 54, configured to delete the recorded data of the deduplication traffic under the multiple time windows when the current counted time of the timing module 53 is greater than or equal to the expiration time threshold.

In an embodiment, the plurality of service time windows include the minimum service time window and the next minimum service time window, and the recording module 52 is further configured to determine the current times of recording of the deduplication traffic in the minimum service time window; and under the condition that the current times is greater than or equal to a time threshold value, judging the deduplication flow as a repeated flow.

In one embodiment, the preset flow configuration information further includes flow rule information, the flow rule information includes a statistical duration, and the apparatus further includes an output module 55, where:

the obtaining module 50 is further configured to obtain the current system time and the statistical duration in the traffic rule information;

the determining module 51 is further configured to determine a statistical time period for the deduplication traffic according to the current system time and the statistical duration;

the obtaining module 50 is further configured to obtain recorded data of all the service time windows in the statistical time period;

and the output module 55 is configured to output alarm information if it is determined that the duplicate removal flow meets the flow rule corresponding to the flow rule information according to the recorded data.

It should be noted that the functions of each functional module of the deduplication traffic recording apparatus described in the embodiment of the present invention may be specifically implemented according to the method in the method embodiment described in fig. 1 or fig. 3, and the specific implementation process may refer to the description related to the method embodiment in fig. 1 or fig. 3, which is not described again here.

Referring to fig. 6, fig. 6 is a schematic block diagram of a server according to an embodiment of the present invention. As shown in fig. 6, the server includes a processor 601, a memory 602, and a network interface 603. The processor 601, the memory 602 and the network interface 603 may be connected by a bus or other means, and the bus connection is taken as an example in fig. 6 in the embodiment of the present invention. Wherein the network interface 603 is controlled by the processor for transceiving messages, the memory 602 is for storing a computer program comprising program instructions, and the processor 601 is for executing the program instructions stored by the memory 602. Wherein the processor 601 is configured to call the program instruction to perform: under the condition of receiving a message, acquiring field point information of the message, and determining whether the flow corresponding to the message is a duplicate removal flow matched with preset flow configuration information or not according to the field point information; if the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information, analyzing the field point information and determining the service generation time and the indication information of the duplication elimination flow, wherein the indication information is used for indicating to execute flow adding operation or flow subtracting operation on the duplication elimination flow; determining a plurality of service time windows for recording the deduplication traffic according to the preset traffic configuration information and the service generation time, wherein each service time window in the plurality of service time windows corresponds to different time granularities; according to the indication information, recording the duplication elimination flow under the service time windows respectively

It should be understood that in the embodiment of the present invention, the Processor 601 may be a Central Processing Unit (CPU), and the Processor 601 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 602 may include both read-only memory and random access memory and provides instructions and data to the processor 601. A portion of the memory 602 may also include non-volatile random access memory. For example, the memory 602 may also store device type information.

In a specific implementation, the processor 601, the memory 602, and the network interface 603 described in this embodiment of the present invention may execute the implementation described in the method embodiment shown in fig. 1 or fig. 3 provided in this embodiment of the present invention, and may also execute the implementation of the deduplication traffic recording apparatus described in this embodiment of the present invention, which is not described herein again.

In another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program comprising program instructions that when executed by a processor implement: under the condition of receiving a message, acquiring field point information of the message, and determining whether the flow corresponding to the message is a duplicate removal flow matched with preset flow configuration information or not according to the field point information; if the flow corresponding to the message is the duplication elimination flow matched with the preset flow configuration information, analyzing the field point information and determining the service generation time and the indication information of the duplication elimination flow, wherein the indication information is used for indicating to execute flow adding operation or flow subtracting operation on the duplication elimination flow; determining a plurality of service time windows for recording the deduplication traffic according to the preset traffic configuration information and the service generation time, wherein each service time window in the plurality of service time windows corresponds to different time granularities; and respectively recording the duplication elimination flow under the plurality of service time windows according to the indication information. The computer readable storage medium may be an internal storage unit of the server according to any of the foregoing embodiments, for example, a hard disk or a memory of the server. The computer readable storage medium may also be an external storage device of the server, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the server. Further, the computer-readable storage medium may also include both an internal storage unit of the server and an external storage device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the server. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

While the invention has been described with reference to a number of embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A deduplication traffic recording method, comprising:

2. The method of claim 1, wherein the predetermined traffic configuration information comprises a recorded time accuracy, and wherein determining a plurality of service time windows for recording the deduplication traffic according to the predetermined traffic configuration information and the service generation time comprises:

3. The method according to claim 2, wherein the indication information is used to indicate that a traffic adding operation is performed on the duplication removal traffic, and the recording the duplication removal traffic under the plurality of service time windows respectively according to the indication information comprises:

4. The method according to claim 2, wherein the indication information is used to indicate that a traffic reduction operation is performed on the duplication removal traffic, and the recording the duplication removal traffic under the plurality of service time windows respectively according to the indication information comprises:

analyzing the indication information, and determining whether the indication information is used for indicating to execute the flow reduction operation on the de-duplicated flow;

and if the indication information is used for indicating that the flow reduction operation is performed on the de-duplication flow, performing the reduction operation on the occurrence frequency of the de-duplication flow under the plurality of service time windows respectively.

5. The method according to claim 1, wherein the preset traffic configuration information further includes a time-to-failure threshold, and after the recording the deduplication traffic under the plurality of traffic time windows respectively, the method further includes:

starting a timing function for the plurality of business time windows;

and deleting the recorded data of the deduplication flow under the plurality of service time windows under the condition that the current timed length is greater than or equal to the failure duration threshold value.

6. The method of claim 3, wherein the plurality of traffic time windows includes the minimum traffic time window and the next smallest traffic time window, and wherein the determining whether the de-duplication traffic is duplicate traffic comprises:

determining the current times of recording of the duplication elimination flow under the minimum service time window;

and under the condition that the current times is greater than or equal to a time threshold value, judging the deduplication flow as a repeated flow.

7. The method according to any one of claims 1 to 6, wherein the preset traffic configuration information further includes traffic rule information, the traffic rule information includes a statistical duration, and after determining whether the traffic corresponding to the packet is a deduplication traffic matched with the preset traffic configuration information according to the field point information, the method further includes:

acquiring the current system time and the statistical duration in the flow rule information;

determining a statistical time period for the de-emphasis flow according to the current system time and the statistical duration;

acquiring the recorded data of all the service time windows in the statistical time period;

and if the duplicate removal flow is determined to meet the flow rule corresponding to the flow rule information according to the recorded data, alarm information is output.

8. A deduplication flow recording apparatus, comprising:

the acquisition module is used for acquiring the field point information of the message under the condition of receiving the message;

the determining module is used for determining whether the flow corresponding to the message is the duplicate removal flow matched with preset flow configuration information or not according to the field point information;

the determining module is further configured to determine that a traffic corresponding to the packet is a duplicate removal traffic matched with preset traffic configuration information, analyze the field point information, and determine a service generation time of the duplicate removal traffic and indication information, where the indication information is used to indicate that a traffic adding operation or a traffic subtracting operation is performed on the duplicate removal traffic;

the determining module is further configured to determine, according to the preset traffic configuration information and the service generation time, a plurality of service time windows for recording the deduplication traffic, where each of the plurality of service time windows corresponds to a different time granularity;

and the recording module is used for respectively recording the duplication elimination flow under the plurality of service time windows according to the indication information.

9. A server, comprising a processor and a storage device, the processor and the storage device being interconnected, wherein the storage device is configured to store a computer program comprising program instructions, and wherein the processor is configured to invoke the program instructions to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-7.