WO2016184316A1

WO2016184316A1 - Data flow limiting method and device

Info

Publication number: WO2016184316A1
Application number: PCT/CN2016/081216
Authority: WO
Inventors: 胡四海
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2015-05-15
Filing date: 2016-05-06
Publication date: 2016-11-24
Also published as: CN106302202A; CN106302202B

Abstract

The present invention provides a data flow limiting method and device. The data flow limiting method comprises: computing a locality sensitive Hash value of received data; computing, according to the locality sensitive Hash value of the data and a locality sensitive Hash value of at least one piece of saved data, a similarity value of the data and the at least one piece of data; and determining, according to the similarity value, whether to save the data or not. The present invention can remove data according to the similarity degree and the difference of the data, and can preferentially remove same data, thereby maximizing diversity of the data subjected to flow limiting.

Description

Data limiting method and device

The present application claims priority to Chinese Patent Application Serial No. No. No. No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Technical field

The present application relates to the field of Internet technologies, and in particular, to a data limiting method and apparatus.

Background technique

Calls between computer systems, for various reasons (insufficient resources, high system pressure, etc.), often face the need for traffic restrictions. The existing current limiting scheme generally includes two types, a random current limiting scheme and a hashing scheme. The random current limiting scheme is usually pure random current limiting. In this scheme, the data removed and retained is completely random, and the diversity of the current limiting data cannot be guaranteed. The Hash scheme calculates the hash value to determine whether the two data are the same, and preferentially removes the same data, but cannot distinguish between the two similar data.

Summary of the invention

The purpose of the present application is to solve at least one of the technical problems in the related art to some extent.

To this end, the first object of the present application is to propose a data current limiting method. The method can remove data according to the similarity and difference of the data, and can preferentially remove the same data, thereby maximizing the diversity of the data after the current limiting.

A second object of the present application is to propose a data current limiting device.

In order to achieve the above object, the data current limiting method of the first aspect of the present application includes: calculating a local sensitive hash value of the received data; and according to the locally sensitive hash value of the data and the saved at least one piece of data. a locally sensitive hash value, a similar value of the data to the at least one piece of data is calculated; and whether the data is saved is determined according to the similarity value.

The data current limiting method of the embodiment of the present application calculates a local sensitive hash value of the received data, and then calculates the data according to the local sensitive hash value of the data and the locally sensitive hash value of the saved at least one piece of data. At least one similarity value of the data, and finally determining whether to save the above data according to the similarity value, so that the data can be removed according to the similarity and difference of the data, and the same data can be preferentially removed, so that the diversity of the data after the current limiting can be achieved. maximize.

In order to achieve the above object, the data current limiting device of the second aspect of the present application includes: a calculation module, Calculating a locally sensitive hash value of the received data, and calculating a similarity of the data to the at least one piece of data according to a locally sensitive hash value of the data and a locally sensitive hash value of the saved at least one piece of data And a determining module, configured to determine whether to save the data according to the similarity value calculated by the computing module.

In the data current limiting device of the embodiment of the present application, the calculation module calculates a local sensitive hash value of the received data, and calculates the above according to the local sensitive hash value of the data and the locally sensitive hash value of the saved at least one piece of data. And the determining module determines whether to save the foregoing data according to the similarity value calculated by the calculating module, so that the data may be removed according to the similarity and difference of the data, and the same data may be preferentially removed, thereby Maximize the diversity of data after current limiting.

The aspects and advantages of the present invention will be set forth in part in the description which follows.

DRAWINGS

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from

1 is a flow chart of an embodiment of a data limiting method of the present application;

2 is a flow chart of another embodiment of a data limiting method of the present application;

3 is a schematic structural diagram of an embodiment of a data current limiting device of the present application;

4 is a schematic structural view of another embodiment of a data current limiting device of the present application.

detailed description

The embodiments of the present application are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are intended to be illustrative only, and are not to be construed as limiting. Rather, the embodiment of the present application includes all changes, modifications, and equivalents falling within the spirit and scope of the appended claims.

1 is a flowchart of an embodiment of a data limiting method of the present application. The data limiting method in this embodiment may be implemented by a data limiting device, where the data limiting device may be disposed between an upstream server and a downstream server, specifically The data limiting device can be integrated in the upstream server or the downstream server to implement the function of limiting the data sent by the upstream server to the downstream server. Alternatively, the data limiting device may be disposed in a separate server or as a separate server, the independent server being located between the upstream server and the downstream server. The function of limiting the data sent by the upstream server to the downstream server is implemented.

As shown in FIG. 1, the data limiting method may include:

Step 101: Calculate a Locality Sensitive Hashing (LSH) value of the received data.

Specifically, the received data is the data sent by the upstream server, and after receiving the data sent by the upstream server, the data limiting device limits the flow and sends the data to the downstream server.

Step 102: Calculate a similarity value between the data and the at least one piece of data according to an LSH value of the data and an LSH value of the saved at least one piece of data.

The at least one piece of data saved may be at least one piece of data that has been saved in the cache, and the cache is a cache opened in the data limiting device or in a server including the data limiting device.

Specifically, calculating a similarity value between the data and the at least one piece of data according to the LSH value of the data and the stored LSH value of the at least one piece of data may be: calculating a difference between an LSH value of the data and an LSH value of the at least one piece of data a value, and calculating a similar value of the above data to the at least one piece of data according to the difference value.

The data current limiting device calculates the similarity value between the data and the at least one piece of data according to the difference value, and may calculate the similarity value between the data and the at least one piece of data according to the formula (1).

Wherein, D _i is a difference value between the LSH value of the data and the LSH value of the at least one piece of data; S _i is a similarity value between the data and the at least one piece of data, i is an integer, and i≥1.

In this embodiment, the difference between the LSH value of the data and the LSH value of the at least one piece of data may be a Hamming distance (Hmming Distance; HD) of the LSH value of the data and the LSH value of the at least one piece of data.

Step 103: Determine whether to save the above data according to the similarity value.

Specifically, the data current limiting device determining whether to save the data according to the similarity value may be: the data current limiting device calculates a pass probability of the data according to a maximum value of the similarity values and a predetermined sampling rate; if the pass probability is greater than or equal to The preset threshold is used to save the above data; and if the pass probability is less than the preset threshold, the data is not saved. The preset threshold may be set according to the implementation requirement and/or the system performance, and the size of the preset threshold is not limited in this embodiment. For example, the preset threshold may be It is 50%.

Specifically, saving the foregoing data may be: storing the foregoing data in the cache. Further, after the data is saved, the data current limiting device may further send the data saved in the cache to the downstream server, so that the data sent by the upstream server is restricted and sent to the downstream server.

The data current limiting device calculates the pass probability of the data according to the maximum value of the similarity value and the predetermined sampling rate, and may calculate the pass probability of the data according to the formula (2).

Where P is the probability of passing the above data; L is a predetermined sampling rate, for example: L can be 75%; S _i is a similar value of the above data and the at least one piece of data, i is an integer, i ≥ 1; Max (S _i ) is the maximum of the above similar values.

In the above embodiment, the data current limiting device calculates an LSH value of the received data, and then calculates a similarity value between the data and the at least one piece of data according to the LSH value of the data and the stored LSH value of the at least one piece of data, and finally according to the foregoing The similarity value determines whether the above data is saved, so that the degree of similarity and difference of the data can be removed, the data can be removed, and the same data can be preferentially removed, so that the diversity of the data after the current limit can be maximized.

The following describes the data current limiting method provided by the present application by taking the e-commerce platform transaction data as an example. Suppose there is a system that requires real-time sampling of transaction data and preserves the diversity of sampled data as much as possible. The predetermined sampling rate is 75%, that is, the current limit is required to remove 25% of the traffic.

Assume that in the order of the serial numbers, the transaction data is as shown in Table 1.

Table 1

As can be seen from Table 1, the data of No.1 and No.4 are exactly the same. Based on the transaction data in Table 1, 8 data, the predetermined sampling rate is 75% (that is, the current limit is 25%), and the actual difference is expected to be the smallest difference. The two pieces of data are: No. 4 (no difference from No. 1) and No. 2 (different from No. 1), that is, data No. 1, No. 3, No. 5, No. 6, No. 7, and No. 8 are retained.

This application uses LSH to make the sampled data diversified as much as possible, and to retain sufficient data difference. It can solve the problem of data loss of random current limiting scheme, and can also solve the Hash scheme. It can only judge similar and cannot judge similar. The problem is that after Hash, the problem of the difference in the original content cannot be preserved.

There are many calculation methods for LSH, such as Jaccard, SimHash or MinHash. The application takes a 64-bit SimHash implementation as an example. The corresponding SimHash values in Table 1 can be as shown in Table 2 (on each bit). 0/1 is 1 bit (Bit) bit, and 1 SimHash value can be stored with 64 bit bits).

Table 2

2 is a flowchart of another embodiment of a data limiting method of the present application. As shown in FIG. 2, the data limiting method may include:

In step 201, the cache space is opened.

The cache space is a cache space opened in the data current limiting device or in the server including the data current limiting device, and is used for buffering the LSH value of the latest N latest data sent by the upstream server. N can be configured according to the actual situation. It is recommended to be the full value within 5 minutes. When the value exceeds 1024, the upper limit is 1024 to ensure that the memory limit is a few K.

In this embodiment, due to the quantity relationship, it can be assumed that N=3, and the data current limiting device first calculates and caches the LSH value for the traffic data according to the sequence number sequence. After the data of the first data flows into the data current limiting device, the cache is as shown in Table 3. .

table 3

缓存Cache
10101111010011111111001011011000110101001001001100110111010000101010111101001111111100101101100011010100100100110011011101000010

Step 202: When new traffic data comes in, calculate an LSH value of the data to be stored in the cache, and calculate a difference value between an LSH value of the data to be stored in the cache and an LSH value of at least one piece of data in the cache.

In the present embodiment, the above difference value is expressed by the Hamming distance HD.

After the data No. 2 flows in, the data current limiting device calculates the LSH value of the No. 2 data, and the LSH value of the No. 2 data can be as shown in Table 4.

Table 4

1010111101001111111101101101100011010100100100110011011101000010

Then, the data current limiting device calculates a difference value from the LSH value of the No. 1 data in the cache, wherein the HD calculation method may be: the number of corresponding bits having different LSH values, that is, comparing the LSH value of the No. 2 data with the No. 1 data. What is the difference in the number of bits of the LSH value, and how many HDs are. Preferably, when computing HD, it can be quickly calculated by XOR.

In this embodiment, the comparison between the LSH value of the No. 2 data and the LSH value of the No. 1 data can be as shown in Table 5.

table 5

As can be seen from Table 5, the LSH value of the No. 2 data is different from the LSH value of the No. 1 data, and thus HD=1 can be obtained.

Step 203: Calculate a similarity value of the data to be cached and the at least one piece of data in the cache according to the difference value.

The larger the HD, the lower the similarity value. In different scenarios, the correspondence between HD and similar values is not fixed. In the 64-bit SimHash scenario, it is tested: when HD=1, the similar accuracy is close to 85%; and when HD=10, the similar accuracy is , less than 30%.

The data current limiting means can then calculate the above similar values according to equation (1).

According to the formula (1), the similarity between the LSH value of the data No. 2 and the LSH value of the No. 1 data can be calculated as: S=0.93.

Step 204: Calculate a pass probability of the data to be cached according to the maximum value of the similarity value and the predetermined sampling rate.

The above pass probability may be calculated according to the formula (2), and the data current limiting device may calculate the pass probability of obtaining the No. 2 data according to the formula (2) to be 5.83%.

Step 205: Determine whether the pass probability is greater than or equal to a preset threshold. If yes, step 206 is performed; if the pass probability is less than the preset threshold, step 207 is performed.

The preset threshold may be set according to the implementation requirement and/or the system performance, and the size of the preset threshold is not limited in this embodiment. However, in this embodiment, the preset threshold is 50% as an example for description.

Step 206: The data to be stored in the cache is stored in the cache, and the process ends.

Step 207: The data to be cached is not stored in the cache, and the process ends.

Since the pass probability of the No. 2 data is 5.83%, which is much lower than 50%, the No. 2 data is not stored in the above cache, and the process is exited.

Next, steps 202 to 207 may be repeated to limit the data of No. 3 to No. 8. Since the No. 2 data is not stored in the cache, the data in the cache is as shown in Table 6.

Table 6

2号后缓存Cache after number 2
10101111010011111111001011011000110101001001001100110111010000101010111101001111111100101101100011010100100100110011011101000010

When the data of No. 3 enters the data current limiting device, the LSH value of the data No. 3 and the HD of the LSH value of the No. 1 data are 10, so that the pass probability of obtaining the No. 3 data is 55.6%, which is greater than 50%, so the No. 3 is obtained. The data is stored in the cache, and the data in the cache is as shown in Table 7.

Table 7

3号后缓存Cache after number 3
10101111010011111111001011011000110101001001001100110111010000101010111101001111111100101101100011010100100100110011011101000010
10111111110001101111101010001000110100001001001000110111011000101011111111000110111110101000100011010000100100100011011101100010

When the No. 4 data enters the data current limiting device, the HD value of the LSH value of the No. 4 data and the LSH value of the No. 1 data is 0, and the LSH value of the No. 4 data and the HD of the LSH value of the No. 3 data are 10. Therefore, the pass probability of obtaining the No. 4 data is 0, so the No. 4 data is not stored in the above cache, and the data in the cache is still as shown in Table 7.

When the No. 5 data enters the data current limiting device, the HD value of the LSH value of the No. 5 data and the LSH value of the No. 1 data is 9, and the LSH value of the No. 5 data and the HD of the LSH value of the No. 3 data are 11. Therefore, the probability of obtaining the data of No. 5 can be calculated to be 50%, so the data of No. 5 is stored in the cache, and the data in the cache is as shown in Table 8.

Table 8

5号后缓存Cache after number 5
10101111010011111111001011011000110101001001001100110111010000101010111101001111111100101101100011010100100100110011011101000010
10111111110001101111101010001000110100001001001000110111011000101011111111000110111110101000100011010000100100100011011101100010
10101111110001101110101001011000110001001011001000110011011000101010111111000110111010100101100011000100101100100011001101100010

Continue with data on the 6th, 7th and 8th, and will not repeat them here.

In the above data limiting method, the data current limiting device can perform data limiting according to the difference in data similarity, and preferentially remove the same data, thereby maximizing the diversity of the data after the current limiting.

3 is a schematic structural diagram of an embodiment of the data current limiting device of the present application. The data current limiting device in this embodiment can implement the process of the embodiment shown in FIG. 1 of the present application. As shown in FIG. 3, the data limiting device can be Including: a calculation module 31 and a determination module 32;

The calculation module 31 is configured to calculate an LSH value of the received data, and calculate a similarity value between the data and the at least one piece of data according to an LSH value of the data and an LSH value of the saved at least one piece of data; wherein, the calculating The module 31 is specifically configured to calculate a difference value between the LSH value of the data and the stored LSH value of the at least one piece of data, and calculate a similarity value between the data and the at least one piece of data according to the difference value. The difference value calculated by the calculation module 31 may be a Hamming distance between the LSH value of the data and the LSH value of the at least one piece of data; specifically, the calculation module 31 may calculate the data and the at least one piece of data according to the formula (1). Similar values.

The determining module 32 is configured to determine whether to save the foregoing data according to the similarity value calculated by the calculating module 31.

The data limiting device may be disposed between the upstream server and the downstream server. Specifically, the data limiting device may be integrated into the upstream server or the downstream server to implement the function of limiting the data sent by the upstream server to the downstream server. Alternatively, the data current limiting device may be disposed in a separate server or as a separate server, and the independent server is located between the upstream server and the downstream server to implement upstream services. The function of the data sent to the downstream server is limited. The data received by the upstream server is the data sent by the upstream server. After receiving the data sent by the upstream server, the data limiting device limits the flow and sends the data to the downstream server.

In the above embodiment, the calculation module 31 calculates the LSH value of the received data, and calculates a similarity value between the data and the at least one piece of data according to the LSH value of the data and the LSH value of the saved at least one piece of data; 32: determining whether to save the foregoing data according to the similarity value calculated by the calculating module 31, so that data can be removed according to the similarity and difference of the data, and the same data can be preferentially removed, thereby maximizing the diversity of the data after the current limiting. .

4 is a schematic structural diagram of another embodiment of the data current limiting device of the present application. Compared with the data current limiting device shown in FIG. 3, the difference is that in the data current limiting device shown in FIG. 4, the determining module 32 can The method includes: a probability calculation sub-module 321 and a deposit sub-module 322;

The probability calculation sub-module 321 is configured to calculate a pass probability of the data according to a maximum value of the similarity values calculated by the calculation module 31 and a predetermined sampling rate; specifically, the probability calculation sub-module 321 can calculate the above according to the formula (2). The probability of passing data.

The storage sub-module 322 is configured to save the foregoing data when the probability of passing by the probability calculation sub-module 321 is greater than or equal to a preset threshold. The preset threshold may be set according to the implementation requirement and/or the system performance, and the size of the preset threshold is not limited in this embodiment. For example, the preset threshold may be 50%. .

Specifically, the saving of the data may be: the depositing sub-module 322 storing the data in the cache. Further, after the data is saved, the data current limiting device may further send the data saved in the cache to the downstream server, so that the data sent by the upstream server is restricted and sent to the downstream server.

The data current limiting device can perform data limiting according to the difference in data similarity, and preferentially remove the same data, thereby maximizing the diversity of the data after the current limiting.

It should be noted that in the description of the present application, the terms "first", "second" and the like are used for descriptive purposes only, and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" is two or more unless otherwise stated.

Any process or method description in the flowchart or otherwise described herein may be understood to include a Modules, segments or portions of code of one or more executable instructions for implementing steps of a particular logical function or process, and the scope of preferred embodiments of the application includes additional implementations, which may not be as shown or The order of discussion includes performing functions in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

It should be understood that portions of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: having logic gates for implementing logic functions on data signals. Discrete logic circuit, ASIC with suitable combination logic gate, Programmable Gate Array (PGA), Field Programmable Gate Array (FPGA).

One of ordinary skill in the art can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, one or a combination of the steps of the method embodiments is included.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as stand-alone products, may also be stored in a computer readable storage medium.

The above mentioned storage medium may be a read only memory, a magnetic disk or an optical disk or the like.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the application. In the present specification, the schematic representation of the above terms does not necessarily mean the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples.

While the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the present application. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

A data current limiting method, comprising:

Calculating a locally sensitive hash of the received data;

Calculating a similarity value of the data and the at least one piece of data according to a locally sensitive hash value of the data and a locally sensitive hash value of the saved at least one piece of data;

Whether to save the data is determined based on the similarity value.
The method according to claim 1, wherein said calculating said data and said at least one piece of data based on a locally sensitive hash value of said data and a locally sensitive hash value of said saved at least one piece of data Similar values include:

Calculating a difference value between the locally sensitive hash value of the data and a locally sensitive hash value of the at least one piece of data;

Calculating a similarity value of the data and the at least one piece of data according to the difference value.
The method according to claim 1 or 2, wherein the determining whether to save the data according to the similarity value comprises:

Calculating a pass probability of the data according to a maximum value of the similarity values and a predetermined sampling rate;

The data is saved if the pass probability is greater than or equal to a preset threshold.
The method of claim 2 wherein the difference between the locally sensitive hash value of the data and the locally sensitive hash value of the at least one piece of data comprises a locally sensitive hash value of the data and the Hamming distance of a locally sensitive hash of at least one piece of data.
A data current limiting device, comprising:

a calculation module, configured to calculate a local sensitive hash value of the received data, and calculate the data and the at least according to the locally sensitive hash value of the data and the locally sensitive hash value of the saved at least one piece of data a similar value for a piece of data;

And a determining module, configured to determine whether to save the data according to the similarity value calculated by the computing module.
The device according to claim 5, characterized in that

The calculating module is specifically configured to calculate a difference value between the locally sensitive hash value of the data and the locally sensitive hash value of the saved at least one piece of data, and calculate the data and the at least one piece according to the difference value Similar values for the data.
The apparatus according to claim 5 or 6, wherein the determining module comprises:

a probability calculation submodule, configured to calculate a pass probability of the data according to a maximum value of the similarity values calculated by the calculation module and a predetermined sampling rate;

And storing the sub-module, when the pass probability calculated by the probability calculation sub-module is greater than or equal to a preset threshold, saving the data.
The device of claim 6 wherein:

The difference value calculated by the calculation module includes a Hamming distance of a locally sensitive hash value of the data and a locally sensitive hash value of the at least one piece of data.