CN109302300B

CN109302300B - Data distribution and processing method, system and computer readable recording medium

Info

Publication number: CN109302300B
Application number: CN201710612889.2A
Authority: CN
Inventors: 马冲
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2017-07-25
Filing date: 2017-07-25
Publication date: 2022-03-15
Anticipated expiration: 2037-07-25
Also published as: CN109302300A

Abstract

The application provides a data distribution method and device, a data processing method and a server, wherein the data processing method is applied to any server connected with the data distribution device, and the data processing method comprises the following steps: receiving an event identifier sent by a data distribution device, and acquiring a duration parameter corresponding to the event identifier; counting the total count value of the event corresponding to the event identifier in the duration parameter; a plurality of event identifications and corresponding count values are stored in the server; judging whether the total count value exceeds a preset processing threshold value, and if so, ignoring partial operation which corresponds to the event and cannot be processed repeatedly; and if not, executing partial operation which corresponds to the event and cannot be repeatedly processed, and accumulating the count value of the event according to the receiving time of the event identifier. By adopting the method and the device, errors caused by user operation can be avoided under the condition that the event data is repeated, and the user experience is improved.

Description

Data distribution and processing method, system and computer readable recording medium

Technical Field

The present application relates to the field of internet data processing technologies, and in particular, to a data distribution method and apparatus, and a data processing method and server.

Background

At present, in the rapid development of electronic commerce, more and more users use the internet to perform various online transactions or to transmit information and the like. And the operation generated by the user through the terminal on the internet is sent to the corresponding server to be executed. For example, an operation of logging in a mailbox by a user sends a user name and a login password to a mailbox server through a terminal, the mailbox server generates an event corresponding to the operation and event data, and the event data comprises data related to the user operation. For another example, when a user orders a certain product in an online mall, the user may also send information of the user name, the recipient address, or the ordered product to a server in the online mall via a terminal, and the server generates a corresponding ordering event, where event data of the ordering event includes information of the user name, the address, the product, and the like.

However, in practical applications, due to network delay or equipment failure or repeated submission by a user, the same ordering operation may be repeatedly sent to a server in an online shopping mall, and in this case, the same ordering operation may be repeatedly sent.

How to process the repeatedly transmitted event data is called a technical problem to be solved urgently.

Disclosure of Invention

The inventor finds that the prior art cannot accurately judge the repeatedly transmitted event data in the research process, according to one aspect of the application, if the generated event data can be counted according to the event identification corresponding to each event data after the server receives the user operation, if the event corresponding to an event identification is repeatedly submitted for multiple times in a short time, the user operation corresponding to the event is not responded, whether the event corresponding to one event data is a repeated event can be accurately judged, and whether the user operation corresponding to the event needs to be responded can be further determined.

Based on this, the application provides a data distribution method and a data processing method, which are used for generating corresponding event identifiers for each event data, and respectively distributing the event data to each server according to the number of the servers which process whether the events are repeated, and each server counts whether the frequency of a certain event identifier stored by the server in a section of event exceeds a preset processing threshold, if so, the operations which cannot be repeatedly processed corresponding to the event are ignored, so that the event data generated by user operation cannot be repeatedly processed, for example, a deduction operation or an integral deduction operation cannot be repeatedly executed, and the user experience is also improved.

The application also provides a data distribution device and a server, which are used for ensuring the realization and the application of the method in practice.

In order to solve the above problem, the present application discloses a data processing method, including:

receiving an event identifier and acquiring a duration parameter corresponding to the event identifier;

counting the total count value of the event corresponding to the event identifier in the duration parameter;

judging whether the total count value exceeds a preset processing threshold value, and if so, ignoring partial operation which is corresponding to the event and cannot be processed repeatedly;

and if the number of the events does not exceed the preset value, executing partial operation which corresponds to the events and cannot be repeatedly processed, and accumulating the count value of the events according to the receiving time of the event identification.

The storage space comprises a timestamp and a segment count value, wherein the timestamp is used for indicating an event corresponding to an event identifier received in the interval duration from the timestamp and needs to be accumulated in the storage space; the accumulating the count value of the event according to the receiving time of the event identifier includes:

calculating a quotient of the receiving time of the event identifier divided by the interval duration as a timestamp to be stored, and calculating a remainder of the receiving time and the number of the storage spaces as an index of the storage spaces to be stored;

judging whether an original timestamp included in a storage space corresponding to the to-be-stored space index is the same as the to-be-stored timestamp, and if so, accumulating a segment count value corresponding to the original timestamp;

and if not, clearing the storage space corresponding to the original timestamp, replacing the original timestamp with the timestamp to be stored, and accumulating the segment count value of the event corresponding to the timestamp to be stored.

Wherein, the counting the total count value of the event corresponding to the event identifier in the duration parameter includes:

acquiring the receiving time of the event identifier;

calculating a quotient of the receiving time and the interval duration and a difference between the quotient and the time stamp included in each storage space, and taking each difference as a query parameter of each storage space;

and judging whether the query parameters of each storage space are smaller than the duration parameters, if so, acquiring the segment count values included in the storage space and accumulating to obtain the total count value of the event identifier.

The embodiment of the application also provides a data distribution method, which comprises the following steps:

receiving event data, the event data comprising: relevant data of the operation of the user on the foreground system;

generating an event identifier corresponding to the event data according to a preset identifier generation rule;

and distributing the event data corresponding to the event identification to the plurality of servers according to the number of the servers.

The generating of the event identifier corresponding to the event data according to the preset identifier generation rule includes:

acquiring a user identifier, an operation type identifier and/or an operation object identifier which are included in the event data;

and combining the user identification, the type identification of the operation and/or the identification of the operation object to obtain the event identification.

Wherein, the allocating the event data corresponding to the event identifier to the plurality of servers according to the number of the servers includes:

for each event data, carrying out hash operation on the corresponding event identifier to obtain a hash value of the event identifier;

performing modular operation on the number of the servers by the hash value to obtain an index value;

and sending the event identifier to a server identified by the index value.

An embodiment of the present application further provides a server, where the server includes:

the event identifier receiving unit is used for receiving the event identifier and acquiring a duration parameter corresponding to the event identifier;

the counting unit is used for counting the total count value of the event corresponding to the event identifier in the duration parameter;

the judging unit is used for judging whether the total count value exceeds a preset processing threshold value;

the first processing unit is used for ignoring partial operation which corresponds to the event and cannot be processed repeatedly under the condition that the result of the judging unit is over;

the second processing unit is used for executing partial operation which corresponds to the event and cannot be processed repeatedly under the condition that the result of the judging unit is not more than the result of the judging unit;

and the accumulation unit is used for accumulating the count value of the event according to the receiving time of the event identifier after the second processing unit is triggered.

The storage space comprises a timestamp and a segment count value, wherein the timestamp is used for indicating an event corresponding to an event identifier received in the interval duration from the timestamp and needs to be accumulated in the storage space; the accumulation unit includes:

the second calculating subunit is configured to calculate a quotient obtained by dividing the receiving time of the event identifier by the interval duration as a timestamp to be stored, and calculate a remainder of the receiving time and the number of the storage spaces as an index of the storage spaces to be stored;

the second judging subunit is configured to judge whether an original timestamp included in the storage space corresponding to the to-be-stored space index is the same as the to-be-stored timestamp;

the second accumulation subunit is configured to accumulate the segment count value corresponding to the original timestamp if the result of the determination subunit is yes;

an emptying subunit, configured to, in a case that the second determining subunit is negative, empty the storage space corresponding to the original timestamp, and replace the original timestamp with the to-be-stored timestamp;

and the third accumulation subunit is used for accumulating the segment count value of the event corresponding to the timestamp to be stored.

Wherein the statistical unit includes:

the second acquisition subunit is used for acquiring the receiving time of the event identifier;

the first calculating subunit is configured to calculate a quotient between the receiving time and the interval duration and a difference between the quotient and a timestamp included in each storage space, and use each difference as a query parameter of each storage space;

the first judging subunit is used for judging whether the query parameter of each storage space is smaller than the duration parameter;

and the first accumulation subunit is used for acquiring the segment count value included in the storage space and accumulating the segment count value to obtain the total count value of the event identifier under the condition that the result of the judgment subunit is yes.

An embodiment of the present application further provides a data distribution apparatus, where the data distribution apparatus includes:

an event data receiving unit configured to receive event data, the event data including: relevant data of the operation of the user on the foreground system;

the generating unit is used for generating an event identifier corresponding to the event data according to a preset identifier generating rule;

and the distribution unit is used for distributing the event data corresponding to the event identification to the plurality of servers according to the number of the servers.

Wherein the generating unit includes:

a first obtaining subunit, configured to obtain a user identifier, a type identifier of an operation, and/or an operation object identifier included in the event data;

and the combination subunit is used for combining the user identifier, the type identifier of the operation and/or the operation object identifier to obtain the event identifier.

Wherein the allocation unit includes:

the hash operation subunit is used for carrying out hash operation on the corresponding event identifier for each event data to obtain a hash value of the event identifier;

the modulo operation subunit is used for performing modulo operation on the number of the servers by using the hash value to obtain an index value;

and the sending subunit is configured to send the event identifier to the server identified by the index value.

An embodiment of the present application also provides a computer-readable medium storing a program for causing a computer to execute data distribution, the process including the steps of:

a receiving step of receiving event data, the event data including: relevant data of the operation of the user on the foreground system;

a generation step of generating an event identifier corresponding to the event data according to a preset identifier generation rule;

Another computer-readable medium is provided in an embodiment of the present application, and the computer-readable medium stores instructions for causing a computer to perform a process, including:

a receiving step, receiving an event identifier and acquiring a duration parameter corresponding to the event identifier;

a counting step, counting the total count value of the event corresponding to the event identifier in the duration parameter;

a judgment step of judging whether the total count value exceeds a preset processing threshold value;

a first processing step, configured to ignore a partial operation that is not repeatable and corresponds to the event, if the determination step obtains the exceeding result;

a second processing step, configured to, when the result obtained in the determining step does not exceed the result obtained in the determining step, perform a partial operation that is not repeatable and corresponds to the event;

and an accumulation step of accumulating the count value of the event according to the reception time of the event identifier after the second processing step.

An embodiment of the present application further provides a computer device, where the computer device includes: a processor, a memory, a network interface and a bus system;

the bus system for coupling together the various hardware components of the computer device;

the network interface is used for realizing communication connection between the computer equipment and at least one other computer equipment;

the memory for storing program instructions and/or data;

the processor is used for reading the instructions and/or data stored in the memory and executing the following operations:

An embodiment of the present application further provides another computer device, where the computer device includes: a processor, a memory, a network interface and a bus system;

the memory for storing program instructions and/or data;

counting the total count value of the event corresponding to the event identifier in the duration parameter; a plurality of event identifications and corresponding count values are stored in the server;

An embodiment of the present application further provides a user interface interaction device, including:

the first interaction module is used for forwarding the first trigger instruction to the processor after receiving the first trigger instruction so that the processor receives event data, wherein the event data comprises: relevant data of the operation of the user on the foreground system;

the second interaction module is used for forwarding the first trigger instruction to the processor after receiving a second trigger instruction so that the processor can generate an event identifier corresponding to the event data according to a preset identifier generation rule;

and the third interaction module is used for forwarding the third trigger instruction to the processor after receiving the third trigger instruction so that the processor distributes the event data corresponding to the event identifier to the plurality of servers according to the number of the servers.

An embodiment of the present application further provides another user interface interaction device, including:

the fourth interaction module is used for forwarding a fourth trigger instruction to the processor after receiving the fourth trigger instruction so that the processor receives an event identifier and acquires a duration parameter corresponding to the event identifier;

a fifth interaction module, configured to receive a fifth trigger instruction, and forward the fifth trigger instruction to the processor, so that the processor counts a total count value of an event corresponding to the event identifier in the duration parameter; a plurality of event identifications and corresponding count values are stored in the server;

a sixth interaction module, configured to receive a sixth trigger instruction, and forward the sixth trigger instruction to a processor, so that the processor determines whether the total count value exceeds a preset processing threshold, and if so, ignores a partial operation that is not repeatable and corresponds to the event; and if the number of the events does not exceed the preset value, executing partial operation which corresponds to the events and cannot be repeatedly processed, and accumulating the count value of the events according to the receiving time of the event identification.

An embodiment of the present application further provides a data processing method, including:

the first computing device generates an event identification, wherein the event identification includes or is based on at least one of: a user ID, a timestamp, an operation code and an operation object ID;

the first computing device sends the event identification to a second computing device, wherein the second computing device performs a deduplication operation based on the event identification.

Compared with the prior art, the method has the following advantages:

in the embodiment of the application, the data distribution device can generate the event identifier corresponding to each event data according to the preset identifier generation rule, and when each event identifier is distributed to the servers, the event identifier is distributed according to the number of the servers, so that the same event identifier is distributed to the same server for processing, errors or omission can not occur when the server counts the counting value of the event identifier, and the event identifier is distributed to each server randomly, so that the problem of data inclination of the event identifier among the servers is avoided to a certain extent.

Furthermore, after each server receives the event identifier, the segment count values corresponding to the event identifier stored in a period of time can be accumulated to obtain the total number corresponding to the event identifier, and then the total number is judged with the preset processing threshold value, so that when the total number exceeds the preset processing threshold value, the user operation corresponding to the event of the event identifier is not responded any more, thereby avoiding the error caused by the user operation under the condition that the event data is repeated, and simultaneously, the user experience is improved as the repeated money deduction or the integral deduction of the user and other related operations are avoided.

Furthermore, because the inside of the server is also divided into a plurality of storage spaces, and each storage space stores the segment count value of the event identifier received in a certain segment of event, the segment count values stored in the storage spaces can be quickly found and accumulated to obtain the total number of the events, thereby ensuring the data processing speed and accuracy.

Of course, it is not necessary for any product to achieve all of the above-described advantages at the same time for the practice of the present application.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.

FIG. 1 is an exemplary diagram of an application scenario of the present application in practical application;

FIG. 2 is an exemplary flow chart of a data distribution method embodiment of the present application;

FIG. 3 is an exemplary flow chart of a data processing method embodiment of the present application;

FIG. 4 is a block diagram of an exemplary configuration of an embodiment of a data distribution apparatus of the present application;

FIG. 5 is a block diagram of an exemplary architecture of an embodiment of a data processing apparatus of the present application;

FIG. 6 is a block diagram of an exemplary architecture of a computing device of the present application;

FIG. 7 is a block diagram of an exemplary architecture for another computer device of the present application;

8 a-8 b are interface schematic diagrams of a user interface interaction device of the present application;

fig. 9a to 9b are interface diagrams of another user interface interaction device of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Fig. 1 is a schematic diagram of an exemplary scenario in practical application of the embodiment of the present application. In which, the user generates an operation in the foreground system 101, such as a menu-down operation, a login operation, or an edit operation on a product, which is called an event. The event data is data content related to the operation, such as a user name for placing an order, an order number, a login password, and the like. The foreground system 101 may send each event data to the data distribution device 102 in response to an event generated by a user operation, distribute each event data by the data distribution device 102, and send an event identifier of each event to each server connected to the data distribution device 102. In fig. 1, the number of servers may be set to N, where N is an integer greater than 1. In the server 103, as shown in the embodiment shown in fig. 3, the server 103 may perform statistics on event identifiers assigned to itself, determine whether a total count value (i.e., a number) of events corresponding to the event identifiers does not meet a threshold, and determine whether to execute or ignore a part of operations corresponding to the event and not being repeatable according to a determination result, thereby implementing deduplication processing of each event, and ensuring that the same event does not repeatedly execute some corresponding operations that cause loss to a user, such as deduction or point deduction.

The data distribution method and the data processing method of the present application are described in detail below with reference to the application scenario shown in fig. 1. Referring to fig. 2, a flowchart of an embodiment of a data distribution method according to the present application is shown, where the embodiment is applicable to the data distribution apparatus shown in fig. 1, and the embodiment may include the following steps:

step 201: receiving event data, the event data may include: data related to the user's operation at the foreground system.

After the user generates an operation in the foreground system, the foreground system transmits event data corresponding to the operation to the data distribution device. The event data may include data related to the operation of the user in the foreground system. For example, after browsing product details on the terminal, the user submits an ordering operation of the product a to the foreground system through the terminal, and the foreground system generates an ordering event for the ordering operation, where the event data of the ordering event may include: the user name, the order number, the order object identifier, the receiving address and the like, and the foreground system sends the event data of the order event to the data distribution device.

Step 202: and generating an event identifier corresponding to the event data according to a preset identifier generation rule.

After the data distribution device receives the event data, the data distribution device may generate an event identifier corresponding to each event data for each event identifier according to a preset identifier generation rule. Specifically, the data distribution device may generate the event identifier corresponding to each event data according to a preset identifier generation rule, for example, a person skilled in the art preset an identifier generation rule as follows: the event data generates unique corresponding event identification according to different principles, and contents such as time, user name, order number and the like can be combined, so that the unique corresponding event identification is generated for the event data.

In a possible implementation manner, in step 202, specifically, a user identifier (which may be a user name), a type identifier of an operation, and/or an operation object identifier included in the event data may be obtained first, and then the user identifier, the type identifier of the operation, and/or the operation object identifier are combined to obtain an event identifier corresponding to each event data. The type of the operation identifies an operation type used for representing an event corresponding to the event data, for example, an operation type such as ordering, logging in, modifying, or deleting. The operand identifies an operand, such as item 156789, that uniquely represents the event to which the event data corresponds, i.e., the item represented by unique correspondence 156789.

Specifically, the event identifier obtained by combining the user identifier, the type identifier of the operation, and the operation object identifier may be in the form of: event identification is type identification of user ID _ operation object identification ID. For example, 12345_ credit type, or 12345_ order _ commodity 1234, and so on. The user ID "12345" is used to uniquely identify a user, the type identifier "discount" or "order placement" of the operation is used to indicate what the type of the user operation is, and the operation object identifier "discount type" or "commodity 1234" is used to uniquely identify an operation object for which the user operation is directed. According to an embodiment of the present application, further, the "discount" corresponds to the number code 001, and the "discount type" corresponds to the number code 7, the event identifier may be 12345_001_ 7.

Of course, the foregoing specific data are merely exemplary data for those skilled in the art to understand the present application, and should not be construed as limiting the embodiments of the present application.

Step 203: and distributing the event data corresponding to the event identification to the plurality of servers according to the number of the servers.

After the data distribution device correspondingly generates each event identifier for each event data, a corresponding server may be distributed for each event, and the event identifier corresponding to each event may be sent to the server distributed for the event identifier. Specifically, the data distribution device may distribute the event identifications according to the following steps a1 to A3:

step A1: and for each event data, carrying out hash operation on the corresponding event identifier to obtain a hash value of the event identifier.

First, the event identifier obtained in step 202 is hashed, for example, by using a hash algorithm such as CRC32 or MD5, to obtain a hash value of each event identifier.

Step A2: and performing modular operation on the number of the servers by using the hash value to obtain an index value.

In order to ensure that the problem of data skew cannot occur between the servers, and the same event identifier is allocated to the same server, in this step, modulo operation is performed on the number of the servers by the hash value of each event identifier to obtain the index value of the server corresponding to each event identifier. For example, assume that there are three servers: server 1, server 2, and server 3, if the index value possibly obtained by performing modulo operation on 3 by the event identifier is 0, 1, or 2, respectively, the event identifier with the index value of 0 is assigned to server 1, the event identifier with the index value of 1 is assigned to server 2, and the event identifier with the index value of 2 is assigned to server 3.

Step A3: and sending the event identifier to a server identified by the index value.

Then, the data distribution device sends the event id to the server corresponding to the index value, for example, all the event ids with index value 0 are sent to server 1, all the event ids with index value 1 are sent to server 2, and all the event ids with index value 2 are sent to server 3.

It can be seen that, in the embodiment of the present application, the data distribution device may generate event identifiers corresponding to the event data according to a preset identifier generation rule, and when distributing the event identifiers to the servers, distribute the event identifiers according to the number of the servers, so as to ensure that the same event identifier is distributed to the same server for processing, so that no error or omission occurs when the server counts the count value of the event identifier, and because the distribution of the event identifier to each server is random, the problem of data skew of the event identifier between the servers is avoided to a certain extent.

In the embodiment of the present application, after the data distribution device sends the event identifier to each server, in order to facilitate statistics of the number of times that the event corresponding to the event identifier in a single time slice is submitted by the user, each server may set a plurality of storage spaces and time slices corresponding to one, respectively. The time intervals of the time slices are the same, each storage space respectively comprises a timestamp and a segment count value, and the timestamp is used for indicating an event corresponding to an event identifier received in the time interval from the timestamp and needs to be accumulated in the storage space to which the timestamp belongs.

For example, for a server, 3 storage spaces are set: the sizes of the stores 1, the stores 2 and the stores 3 may be different, the interval durations interval of the time slices corresponding to the stores are the same, and the interval may be in the order of milliseconds, for example, the interval is 2 milliseconds. Each store includes a timestamp and a segment count value, and the value ts recorded by the timestamp can be used to indicate an event in the period from ts to ts + interval, and its corresponding event identifier should be accumulated to the segment count value included in the storage space to which the timestamp belongs. The segment count value may be implemented by a mapping table, that is, the segment count value of the received event identifier corresponding to each storage space may be stored in a mapping table, where the event identifier and the segment count value thereof in a period from ts to ts + interval are stored in the mapping table.

Assume that there are 3 stores 1, store2, and store3, corresponding to 3 time slices: "2017/5/900: 00: 01", "2017/5/900: 00: 02" and "2017/5/900: 00: 03", i.e., store1 holds the count value of event identifications entered during the "2017/5/900: 00: 01" period, and so on, store3 holds the count value of event identifications entered during the "2017/5/900: 00: 03" period, i.e., each storage space stores 1 second of data, i.e., interval is 1000 (milliseconds). Assume that the receiving time instants corresponding to the 3 time slices are: 1494259201000 (milliseconds), 1494259202000 (milliseconds) and 1494259203000 (milliseconds). The timestamp corresponding to the storage space store is the quotient of the receiving time and the interval duration, that is, the timestamp ts1 corresponding to the storage space store1 is 1494259201000/interval 1494259201, the timestamp ts2 corresponding to the storage space store2 is 1494259202000/interval 1494259202, and the timestamp ts3 corresponding to the storage space store3 is 1494259203000/interval 1494259203.

In the embodiment of the application, the n storage spaces in the server can be recycled. Still taking the example of 3 storage spaces corresponding to 3-second time slices, respectively, store1 stores the segment count value of the event marker received in the first second, store2 stores the segment count value of the event marker received in the second, store3 stores the segment count value of the event marker received in the third second, and so on, at the 4 th second, store the segment count value of the event marker received in the 4 th second by first storage space store1, at the 5 th second, store the segment count value of the event marker received in the 5 th second by second storage space store2, and so on.

Referring to fig. 3, a flowchart of an embodiment of a data processing method according to the present application is shown, where the processing method is applied to any server 103 connected to a data distribution device, and the embodiment may include the following steps 301 to 306:

step 301: and receiving an event identifier and acquiring a duration parameter corresponding to the event identifier.

After the data distribution device sends each event identifier to each server according to the index value, for each server, after receiving the event identifier, the time length parameter corresponding to the event identifier may be acquired, so that the server may count the count value of the event corresponding to the event identifier in the time range indicated by the time length parameter. The duration parameter may be configured in advance by those skilled in the art and stored in each server, and for each type of event, one duration parameter may be set to represent the time range of all events of the type. For example, for an event of the type of placing an order, a count value within 2 seconds may be queried, and for an event of the type of placing an order, the duration parameter is 2 seconds. Of course, other values may be preset for the duration parameter by one skilled in the art.

Step 302: and counting the total count value of the event corresponding to the event identification in the duration parameter.

Because the same event identifier is distributed by the data distribution device and sent to the same server, the server only needs to count the total count value corresponding to the received event identifier, and other servers do not need to be considered.

Specifically, according to the time stamp corresponding to each storage space included in the server, the storage space and the segment count value recorded in which storage spaces need to be counted are determined, and then the segment count values in each storage space to be counted are accumulated to obtain the total count value of the event identifier. One implementation of step 302 may include steps B1-B4:

step B1: and acquiring the receiving time of the event identifier.

Firstly, the receiving time of the event identifier A to be counted is determined. Assuming that the event identifier a is received by the time server 1 of 5/9/2017, 00:00:03, the corresponding reception time is 1494259203000.

Step B2: and calculating the quotient of the receiving time and the interval duration and the difference between the quotient and the time stamp included in each storage space, and taking each difference as the query parameter of each storage space.

In this step, the quotient of the calculated reception time "2017/5/900: 00: 03" and the interval duration is 1494259203000/interval 1494259203, and the quotient "1494259203" is subtracted from the timestamps 1494259201, 1494259202, and 1494259203 corresponding to the store1, the store2, and the store3, respectively, to obtain three differences: 2. 1 and 0, i.e., the query parameter for store1 is 2, the query parameter for store2 is 1, and the query parameter for store3 is 0.

Step B3: and judging whether the query parameter of each storage space is smaller than the duration parameter, and if so, entering the step B4.

For the 3 storage spaces, whether the corresponding query parameters are smaller than the value of the duration parameter is respectively judged, that is, whether the query parameters corresponding to the store1, the store2 and the store3 are smaller than 2 is judged, the query parameters of the store2 and the store3 are smaller than 2, and the query parameter of the store1 is equal to 2.

Step B4: and acquiring the segmented count values included in the storage space and accumulating to obtain the total count value of the event identifier.

Then, for the store2 and the store3, the segment count values corresponding to the event identifier a recorded in the mapping tables in the two storage spaces are accumulated to obtain the total count value of the event identifier a. For example, if the segment count value of the event identifier a in the mapping table of the store2 is 2, and the segment count value of the event identifier a in the mapping table of the store3 is 1, the total count value of the event identifier a in 2 seconds of the duration parameter is 2+1 — 3.

Step 303: judging whether the total count value exceeds a preset processing threshold value, if so, entering a step 304; if not, step 305 is entered.

Next, it is determined whether the total count value obtained in step 302 exceeds a preset processing threshold, for example, the processing threshold may be preset to 2, and the step 304 is executed if the total count value of the event identifier a exceeds the preset processing threshold.

Step 304: and ignoring part of operations which are not repeatedly processed and correspond to the events.

In practical applications, operations corresponding to some events are not repeatable, for example, a payment request when a user pays for the same order, if the payment request is repeatedly submitted due to reasons such as web page delay, repeated deduction operations are performed on the user, and the repeated operations will bring bad influence to user experience. Therefore, in this embodiment, for the operation corresponding to the event of this type, for example, an operation of deducting the user's credit, or a deduction operation, and an operation of disabling the user's account, etc., a corresponding processing threshold may be preset, and when the total count value of the event identifier of the event is found to exceed the preset processing threshold, that part of the operation, which is not repeatedly processed, corresponding to the event, for example, an operation of deducting the credit or deduction, may be directly ignored.

Step 305: and executing the part of the operation which is not repeatedly processed and corresponds to the event, and then entering step 306.

And if the total count value corresponding to the event identifier does not exceed the preset processing threshold, the part of the operation, which is not capable of being processed repeatedly, of the part related to the private information of the user corresponding to the event can be executed.

Step 306: and accumulating the count value of the event according to the receiving time of the event identifier.

And then, according to the receiving time of the event identifier, accumulating the count value of the event, namely adding 1 to the segment count value corresponding to the event identifier in the storage space corresponding to the receiving time. Specifically, the step 306 may include steps C1 to C4:

step C1: and calculating the quotient of the receiving time of the event identifier divided by the interval duration to serve as a timestamp to be stored, and calculating the remainder of the receiving time and the number of the storage spaces to serve as an index of the storage spaces to be stored.

Firstly, when an event identifier is received, calculating a quotient of the receiving time of the event identifier divided by the interval duration as a timestamp to be stored, namely ts is time/interval; and calculating the remainder of the number of the storage spaces at the receiving moment as an index of the space to be stored, namely index is time/n, wherein n is the number of the storage spaces. Still, taking 3 storage spaces as an example in the server, assuming that the obtained index is 0, it indicates that 1 needs to be added to the segment count value of the event identifier in the first storage space store1, and so on, and assuming that the obtained index is 2, it indicates that 1 needs to be added to the segment count value of the event identifier in the storage space store 3.

Step C2: judging whether the original time stamp included in the storage space corresponding to the to-be-stored space index is the same as the to-be-stored time stamp or not, and if so, entering the step C3; if not, step C4 is entered.

Next, it is determined whether the original timestamp included in the storage space corresponding to the index obtained in step C1 is the same as the timestamp to be stored actually calculated in step C1, and if so, the process proceeds to step C3, and 1 is added to the segment count value corresponding to the event identifier stored in the mapping table of the storage space. For example, assuming that the obtained index is zero, i.e. the first storage space store1 is the space to be stored, the time stamp ts1 included in the store1 is the original time stamp, and if the time stamp calculated in step C1 is also equal to ts1, step C3 is performed, and if not equal to ts1, step C4 is performed.

Step C3: and accumulating the segment count value corresponding to the original timestamp.

In this step, 1 is added to the segment count value of the event identifier in the mapping table in the storage space corresponding to the index of the space to be stored, which indicates that an event corresponding to the event identifier is received at a time, so that when it is necessary to determine whether to execute a part of operations corresponding to the event and not repeatedly executable in the following process, the segment count values are accumulated to obtain a total count value.

Step C4: and clearing the storage space corresponding to the original timestamp, replacing the original timestamp with the timestamp to be stored, and accumulating the segment count value of the event corresponding to the timestamp to be stored.

If not, the index to be stored is expired, then the mapping table in the storage space corresponding to the index to be stored is cleared, the time stamp of the cleared storage space is set to the time stamp ts calculated in the step C1, the segment count value corresponding to the event identifier is added by 1, and the segment count value is stored in the mapping table in the storage space.

Therefore, in the embodiment of the application, after the server receives the event identifier, the segment count values corresponding to the event identifier stored in a period of time can be accumulated to obtain the total number corresponding to the event identifier, and then the total number is judged with the preset processing threshold, so that when the total number exceeds the preset processing threshold, the user operation corresponding to the event identifier is not responded any more, and thus, data errors caused by executing the user operation corresponding to the event under the condition that the event data is repeated are avoided.

For simplicity of explanation, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art will appreciate that the present application is not limited by the order of acts, as some steps may occur in other orders or concurrently with other steps based on the disclosure herein. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

Corresponding to the method provided by the foregoing embodiment of the data distribution method in the present application, referring to fig. 4, the present application further provides an embodiment of a data distribution apparatus, in which the data distribution apparatus is connected to a plurality of servers, and the data distribution apparatus may include:

an event data receiving unit 401, configured to receive event data, where the event data includes: data related to the user's operation at the foreground system.

A generating unit 402, configured to generate an event identifier corresponding to the event data according to a preset identifier generation rule.

The generating unit 402 may specifically include:

a first obtaining subunit, configured to obtain a user identifier, a type identifier of an operation, and/or an operation object identifier included in the event data; and the combination subunit is used for combining the user identifier, the type identifier of the operation and/or the operation object identifier to obtain the event identifier.

The allocating unit 403 is configured to allocate the event data corresponding to the event identifier to the servers according to the number of the servers.

The allocating unit 403 may specifically include:

the hash operation subunit is used for carrying out hash operation on the corresponding event identifier for each event data to obtain a hash value of the event identifier; the modulo operation subunit is used for performing modulo operation on the number of the servers by using the hash value to obtain an index value; and the sending subunit is used for sending the event identifier to the server identified by the index value.

Corresponding to the method provided by the above-mentioned data processing method embodiment of the present application, referring to fig. 5, the present application further provides an embodiment of a server, and in this embodiment, the server is connected to the data distribution apparatus, and the server may include:

an event identifier receiving unit 501, configured to receive an event identifier sent by the data distribution apparatus, and obtain a duration parameter corresponding to the event identifier.

A counting unit 502, configured to count a total count value of the event corresponding to the event identifier within the duration parameter; the server stores a plurality of event identifications and corresponding count values.

The statistical unit 502 may specifically include:

the second acquisition subunit is used for acquiring the receiving time of the event identifier; the first calculating subunit is configured to calculate a quotient between the receiving time and the interval duration and a difference between the quotient and a timestamp included in each storage space, and use each difference as a query parameter of each storage space; the first judging subunit is used for judging whether the query parameter of each storage space is smaller than the duration parameter; and the first accumulation subunit is configured to, if the result of the first judgment subunit is yes, obtain the segment count values included in the storage space, and accumulate the segment count values to obtain a total count value of the event identifier.

A determining unit 503, configured to determine whether the total count value exceeds a preset processing threshold.

A first processing unit 504, configured to ignore a partial operation of unrepeatable processing corresponding to the event if the result of the determining unit is the excess.

And a second processing unit 505, configured to execute, when the result of the determining unit is not exceeded, a partial operation of unrepeatable processing corresponding to the event.

And an accumulating unit 506, configured to accumulate the count value of the event according to the receiving time of the event identifier after the second processing unit is triggered.

The server comprises a plurality of one-to-one corresponding storage spaces and time slices, the interval duration of each time slice is the same, each storage space comprises a timestamp and a segment count value, and the timestamp is used for indicating an event corresponding to an event identifier received in the interval duration from the timestamp and needs to be accumulated in the storage space; the accumulation unit 506 may specifically include:

the second calculating subunit is configured to calculate a quotient obtained by dividing the receiving time of the event identifier by the interval duration as a timestamp to be stored, and calculate a remainder of the receiving time and the number of the storage spaces as an index of the storage spaces to be stored; the second judging subunit is configured to judge whether an original timestamp included in the storage space corresponding to the to-be-stored space index is the same as the to-be-stored timestamp; the second accumulation subunit is configured to accumulate the segment count value corresponding to the original timestamp if the result of the second judgment subunit is yes; an emptying subunit, configured to, in a case that the second determining subunit is negative, empty the storage space corresponding to the original timestamp, and replace the original timestamp with the to-be-stored timestamp; and the third accumulation subunit is used for accumulating the segment count value of the event corresponding to the timestamp to be stored.

An embodiment of the present application also provides a computer-readable medium storing a program for causing a computer to execute data distribution, where the process may include the steps of:

a generation step of generating an event identifier corresponding to the event data according to a preset identifier generation rule; and the number of the first and second groups,

Another computer-readable medium is provided in an embodiment of the present application, and the computer-readable medium stores instructions for causing a computer to perform the following steps:

a second processing step, configured to, when the result obtained in the determining step does not exceed the result obtained in the determining step, perform a partial operation that is not repeatable and corresponds to the event; and the number of the first and second groups,

Fig. 6 is a schematic hardware structure diagram of a computer device 600 according to an embodiment of the present application. The computer device 600 may be used to implement data distribution, and may be a client, for example. I.e. the computer device 600 may be used to perform the methods provided in the above embodiments. In this embodiment, the computer apparatus 600 includes: a processor 601, a memory 602, a network interface 603, and a bus system 604.

The bus system 604 is used to couple the various hardware components of the computing device 600 together.

The network interface 603 is used to implement communication connection between the computing device 600 and at least one other computing device, and may use the internet, a wide area network, a local area network, a metropolitan area network, or the like.

The memory 602 is used to store program instructions and/or data.

The processor 601 is configured to read instructions and/or data stored in the memory 602, and perform the following operations:

receiving event data, the event data comprising: relevant data of the operation of the user on the foreground system; generating an event identifier corresponding to the event data according to a preset identifier generation rule; and distributing the event data corresponding to the event identification to the plurality of servers according to the number of the servers.

The generating an event identifier corresponding to the event data according to a preset identifier generation rule may include: acquiring a user identifier, an operation type identifier and/or an operation object identifier which are included in the event data; and combining the user identification, the type identification of the operation and/or the identification of the operation object to obtain the event identification.

The allocating, according to the number of servers, the event data corresponding to the event identifier to the multiple servers may include:

for each event data, carrying out hash operation on the corresponding event identifier to obtain a hash value of the event identifier; performing modular operation on the number of the servers by the hash value to obtain an index value; and sending the event identifier to a server identified by the index value.

Fig. 7 is a schematic hardware configuration diagram of another computer device 700 according to an embodiment of the present application. The computer device 700 may be used to implement data distribution, and may be a client, for example. I.e. the computer device 700 may be used to perform the methods provided in the above embodiments. In this embodiment, the computer device 700 includes: a processor 701, a memory 702, a network interface 703 and a bus system 704.

The bus system 704 is used to couple the various hardware components of the computing device 700 together.

The network interface 703 is used to implement a communication connection between the computing device 700 and at least one other computing device, and may use the internet, a wide area network, a local area network, a metropolitan area network, or the like.

The memory 702 is used to store program instructions and/or data.

The processor 701 is configured to read instructions and/or data stored in the memory 702, and perform the following operations:

judging whether the total count value exceeds a preset processing threshold value, and if so, ignoring partial operation which is corresponding to the event and cannot be processed repeatedly; and if the number of the events does not exceed the preset value, executing partial operation which corresponds to the events and cannot be repeatedly processed, and accumulating the count value of the events according to the receiving time of the event identification.

The server comprises a plurality of one-to-one corresponding storage spaces and time slices, the interval duration of each time slice is the same, each storage space comprises a timestamp and a segment count value, and the timestamp is used for indicating an event corresponding to an event identifier received in the interval duration from the timestamp and needs to be accumulated in the storage space; the accumulating the count value of the event according to the receiving time of the event identifier may include:

calculating a quotient of the receiving time of the event identifier divided by the interval duration as a timestamp to be stored, and calculating a remainder of the receiving time and the number of the storage spaces as an index of the storage spaces to be stored; judging whether an original timestamp included in a storage space corresponding to the to-be-stored space index is the same as the to-be-stored timestamp, and if so, accumulating a segment count value corresponding to the original timestamp; and if not, clearing the storage space corresponding to the original timestamp, replacing the original timestamp with the timestamp to be stored, and accumulating the segment count value of the event corresponding to the timestamp to be stored.

The counting the total count value of the event corresponding to the event identifier in the duration parameter may include:

acquiring the receiving time of the event identifier; calculating a quotient of the receiving time and the interval duration and a difference between the quotient and the time stamp included in each storage space, and taking each difference as a query parameter of each storage space; and judging whether the query parameters of each storage space are smaller than the duration parameters, if so, acquiring the segment count values included in the storage space and accumulating to obtain the total count value of the event identifier.

In practical applications, the data processing method according to the embodiment of the present application may be implemented by first generating, by a first computer device, an event identifier, where the event identifier includes or is based on at least one of the following data: user ID, time stamp, operation code, operation object ID; and then the first computer device sends the event identification to a second computer device, wherein the second computer device carries out the duplicate removal operation based on the event identification.

Referring to fig. 8a to 8b, the present application further provides a user interface interaction device, where the interaction device may include:

the first interaction module is used for forwarding the first trigger instruction to the processor after receiving the first trigger instruction so that the processor receives event data, wherein the event data comprises: data related to the user's operation at the foreground system.

After the user clicks the control of "receiving event data" (refer to fig. 8a) through the interactive interface, the first interaction module of the interaction device receives the first trigger instruction, and then the first interaction module forwards the first trigger instruction to the processor, so that the processor may receive event data of an event sent by the foreground system, where the event data may include: data related to the user's operation at the foreground system.

And the second interaction module is used for receiving a second trigger instruction and forwarding the first trigger instruction to the processor so as to receive an event identifier corresponding to the event data generated by the processor according to a preset identifier generation rule.

After the user clicks the control (refer to fig. 8a) of the "generate event identifier" through the interactive interface, the second interaction module of the interaction device receives the second trigger instruction, and then the second interaction module forwards the second trigger instruction to the processor, so as to generate the event identifier corresponding to the event data according to the preset identifier generation rule.

And a third interaction module, configured to receive a third trigger instruction, and forward the third trigger instruction to the processor, so that the processor allocates event data corresponding to the event identifier to the servers according to the number of the servers (see fig. 8 b).

After the user clicks a control (refer to fig. 8a) of "display distribution result" through the interactive interface, the third interaction module of the interaction device receives a third trigger instruction, and then the third interaction module forwards the third trigger instruction to the processor, so that the event data corresponding to the event identifier is distributed to the plurality of servers according to the number of the servers, and receives the distribution result sent by the processor for display.

The functions described in the method of the present embodiment, if implemented in the form of software functional units and sold or used as independent products, may be stored in a storage medium readable by a computing device. Based on such understanding, part of the contribution to the prior art of the embodiments of the present application or part of the technical solution may be embodied in the form of a software product stored in a storage medium and including several instructions for causing a computing device (which may be a personal computer, a server, a mobile computing device or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Referring to fig. 9a to 9b, the present application further provides another user interface interaction device, where the interaction device may include:

and the fourth interaction module is used for forwarding the fourth trigger instruction to the processor after receiving the fourth trigger instruction so that the processor receives the event identifier and acquires the duration parameter corresponding to the event identifier.

After the user clicks the control of "receiving the event identifier" (refer to fig. 9a) through the interactive interface, the fourth interaction module of the interaction device receives the fourth trigger instruction, and then the fourth interaction module forwards the fourth trigger instruction to the processor, so that the processor can receive the event identifier sent by the data distribution device and acquire the duration parameter corresponding to the event identifier.

A fifth interaction module, configured to receive a fifth trigger instruction, and forward the fifth trigger instruction to the processor, so as to receive a total count value of an event corresponding to the event identifier in the duration parameter, which is counted by the processor; the server stores a plurality of event identifications and corresponding count values.

After the user clicks the control of the "statistical count value" (refer to fig. 9a) through the interactive interface, the fifth interaction module of the interaction device receives the fifth trigger instruction, and then the fifth interaction module forwards the fifth trigger instruction to the processor, so as to generate the event identifier corresponding to the event data according to the preset identifier generation rule.

A sixth interaction module, configured to receive a sixth trigger instruction, and forward the sixth trigger instruction to a processor, so that the processor determines whether the total count value exceeds a preset processing threshold, and if so, ignores a partial operation that is not repeatable and corresponds to the event; if not, executing the partial operation which is corresponding to the event and can not be repeatedly processed, and accumulating the count value of the event according to the receiving time of the event identifier (refer to fig. 9 b).

After the user clicks a control (refer to fig. 9a) of "display processing result" through the interactive interface, the sixth interaction module of the interaction device receives the sixth trigger instruction, and then the sixth interaction module forwards the sixth trigger instruction to the processor, so that the event data corresponding to the event identifier is distributed to the plurality of servers according to the number of the servers, and the distribution result sent by the processor is received for display.

It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The data distribution method and apparatus, the data processing method and the server provided by the present application are introduced in detail above, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method for data distribution and processing, the method comprising:

performing, by the data distribution apparatus, the steps of:

distributing the event data corresponding to the event identification to a plurality of servers according to the number of the servers;

performing, by the server, the steps of:

2. The method according to claim 1, wherein the generating an event identifier corresponding to the event data according to a preset identifier generation rule includes:

3. The method according to claim 1, wherein the distributing the event data corresponding to the event identifier to the plurality of servers according to the number of the servers comprises:

and sending the event identifier to a server identified by the index value.

4. A data distribution and processing system, characterized in that the data distribution and processing system comprises a data distribution device and a server:

the data distribution apparatus includes:

the distribution unit is used for distributing the event data corresponding to the event identification to a plurality of servers according to the number of the servers;

the server includes:

5. A computer-readable recording medium storing a program for causing a data distribution and processing system including a data distribution apparatus and a server to execute, for data distribution and processing, the data distribution apparatus executing, when the program is executed, the program:

an event data receiving step of receiving event data including: relevant data of the operation of the user on the foreground system;

distributing step, distributing the event data corresponding to the event identification to a plurality of servers according to the number of the servers;

the server performs:

an event identifier receiving step, which receives an event identifier and acquires a duration parameter corresponding to the event identifier;