CN111259045A

CN111259045A - Data processing method, device, server and medium

Info

Publication number: CN111259045A
Application number: CN202010054694.2A
Authority: CN
Inventors: 杨志华; 熊维; 余宝春
Original assignee: Kingdom Financial Nanjing Technology Co ltd
Current assignee: Kingdom Financial Nanjing Technology Co ltd
Priority date: 2020-01-17
Filing date: 2020-01-17
Publication date: 2020-06-09
Anticipated expiration: 2040-01-17
Also published as: CN111259045B

Abstract

The application is applicable to the technical field of data processing, and provides a data processing method, a device, a server and a medium, wherein the method comprises the following steps: receiving a data summarizing instruction sent by a central server; grouping and summarizing the data stored in the node servers according to the data summarizing instruction, and reporting the summarized data to the central server; receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers; and processing the data stored by the node server according to the critical value. By the method, the data processing efficiency can be improved, and the storage space can be saved.

Description

Data processing method, device, server and medium

Technical Field

The present application belongs to the field of data processing technologies, and in particular, to a data processing method, apparatus, server, and medium.

Background

In the field of computer technology, it is often necessary to process large amounts of data. Such as data acquisition, data sorting, and data screening. When the amount of data to be processed is large, the whole data processing process consumes much resources and takes a long time.

Distributed computing can split a large problem into multiple sub-problems, and then solve the sub-problems at each sub-node, thereby shortening the time for solving the whole large problem. However, although data can be collected by using distributed computing at present, when data sorting and screening are performed, a large amount of data needs to be processed in a centralized manner, and all data are processed sequentially by using the same computing device, which requires a large amount of network transmission and storage resources, and the computing time is long.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device, a server and a medium, and can solve the problems of more consumed resources and low efficiency of data processing.

In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a node server, and the method includes:

receiving a data summarizing instruction sent by a central server;

grouping and summarizing the data stored in the node servers according to the data summarizing instruction, and reporting the summarized data to the central server;

receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers;

and processing the data stored by the node server according to the critical value.

In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a central server, and the method includes:

sending a data summarization instruction to a plurality of node servers associated with the central server;

receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping and summarizing stored data by each node server;

determining a critical value according to summarized data returned by the plurality of node servers;

and sending the critical value to the plurality of node servers to instruct the plurality of node servers to process the stored data according to the critical value.

In a third aspect, an embodiment of the present application provides a data processing apparatus, which is applied to a node server, and the apparatus includes:

the instruction receiving module is used for receiving a data summarizing instruction sent by the central server;

the data summarization module is used for grouping and summarizing the data stored in the node server according to the data summarization instruction and reporting the summarized data to the central server;

the critical value receiving module is used for receiving a critical value sent by the central server, and the critical value is determined by the central server according to summarized data of the plurality of node servers;

and the data processing module is used for processing the data stored in the node server according to the critical value.

In a fourth aspect, an embodiment of the present application provides a data processing apparatus, which is applied to a central server, and the apparatus includes:

the instruction sending module is used for sending a data summarizing instruction to a plurality of node servers associated with the central server;

the data receiving module is used for receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, and the summarized data is obtained by grouping and summarizing the stored data by each node server;

the critical value determining module is used for determining a critical value according to summarized data returned by the plurality of node servers;

and the critical value sending module is used for sending the critical value to the plurality of node servers so as to instruct the plurality of node servers to process the stored data according to the critical value.

In a fifth aspect, an embodiment of the present application provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the method according to the first aspect or the second aspect when executing the computer program.

In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when executed by a processor, the computer program implements the method according to the first or second aspect.

In a seventh aspect, an embodiment of the present application provides a computer program product, which when run on a server, causes the server to perform the method of any one of the first aspect or the second aspect.

Compared with the prior art, the embodiment of the application has the advantages that: in the embodiment of the application, a central server is associated with a plurality of node servers, each node server receives a data summarizing instruction sent by the central server, then groups and summarizes data stored by the node servers according to the data summarizing instruction, and reports the summarized data to the central server; the central server determines a critical value according to the summarized data of the plurality of node servers, the node servers receive the critical value sent by the central server, and then the data stored by the node servers are processed according to the critical value. In the data processing process, all original data do not need to be sent to the central server, but the node servers gather the data and then send the gathered data to the central server, so that the data volume needing to be transmitted is reduced, and the storage space of the central server is saved. According to the embodiment, the node server and the central server are used for processing data together, so that the data processing efficiency is improved, and the calculation time is shortened.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a data processing method according to a second embodiment of the present application;

fig. 3 is a schematic flowchart of a data processing method according to a third embodiment of the present application;

fig. 4 is a schematic flowchart of a data processing method according to a fourth embodiment of the present application;

fig. 5 is a schematic flowchart of a data processing method according to a fifth embodiment of the present application;

fig. 6 is a schematic diagram illustrating a relationship between a central server and a node server according to a sixth embodiment of the present application;

fig. 7 is a schematic flowchart of a data processing method according to a sixth embodiment of the present application;

fig. 8 is a schematic structural diagram of a data processing apparatus according to a seventh embodiment of the present application;

fig. 9 is a schematic structural diagram of a data processing apparatus according to an eighth embodiment of the present application;

fig. 10 is a schematic structural diagram of a server according to a ninth embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.

Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application, and as shown in fig. 1, the method includes:

s101, receiving a data summarizing instruction sent by a central server;

in the embodiment of the application, the central server is associated with a plurality of node servers, the central server can communicate with the node servers, and each node server can independently execute tasks. The execution main body of the embodiment is a node server, and the node server is a server capable of performing data screening and data transmission, and comprises a cloud server.

Specifically, when data processing is required, the central server sends a data summarization instruction to the node servers, and the node servers receive the data summarization instruction sent by the central server and perform subsequent data processing according to the data summarization instruction.

S102, grouping and summarizing the data stored in the node server according to the data summarizing instruction, and reporting the summarized data to the central server;

specifically, the node server stores a large amount of raw data, which may include data generation time, data value, order number, customer information, and the like. And the node server groups the stored original data according to a certain characteristic according to the data summarizing instruction. For example, the grouping may be by time period, order number, or data value. And then adding the data values corresponding to all the data of each group to obtain summarized data corresponding to each group, and sending the summarized data of each group to the central server. And after receiving the data summarization instruction, the plurality of node servers associated with the central server all send summarized data to the central server.

S103, receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers;

the threshold value may be a criterion for data screening. For example, the threshold may be a data value, and if the data value of the raw data is greater than the threshold, the raw data is the selected data to be processed; the threshold may be a time value, and if the generation time of the raw data is earlier than the threshold, the raw data is the selected data to be processed; the threshold may be an order number, and the raw data is the selected data to be processed if the order number of the raw data is before the threshold.

Specifically, after receiving the summarized data sent by each node server, the central server sorts the summarized data, determines a critical value, and sends the critical value to each node server through a data bus, and the node server receives the critical value.

And S104, processing the data stored in the node server according to the critical value.

Each node server can independently execute tasks, so that a large amount of raw data can be generated in the process of processing the tasks, and the data can be processed as required. Illustratively, each node server receives an order of a user, then needs to confirm the order meeting the conditions according to a rule specified by the central server, and sends successful purchase applying information to the user corresponding to the order meeting the conditions.

Specifically, the node server screens out required data in the stored original data according to a critical value determined by the central server, and then processes the data according to a preset processing mode. Illustratively, if the data is a user subscription order, a plurality of user subscription orders are generated on each node server, and the total amount of all subscription orders is greater than the preset total amount, the data needs to be screened to determine a user who succeeds in subscription, and then the user subscription order which succeeds in subscription is processed. In the data screening, the order for subscription may be selected according to different conditions, for example, an order with a large amount of money may be selected to be prioritized. The critical value selected by the central server is a value of the amount, and after receiving the critical value, the node server can select the orders with the amount of the stored orders being greater than or equal to the critical value, and then send successful purchase applying information to the users corresponding to the orders.

In this embodiment, the data is stored in each node server, and all the data does not need to be sent to the server, but the node servers gather the data and send the gathered data to the central server, so that the storage space of the central server is saved, and the pressure of data transmission is reduced. The data screening is completed by the central server and the node servers together, and all the node servers can execute in parallel, so that the data processing efficiency is improved, and the data processing time is saved.

Fig. 2 is a schematic flowchart of a data processing method provided in the second embodiment of the present application, and as shown in fig. 2, the method includes:

s201, receiving a data summarizing instruction sent by a central server;

the execution main body of this embodiment is a node server, the node server is associated with a central server, and the data summarization instruction is sent to the node server by the central server and may carry packet information.

S202, grouping the data stored in the node server according to the grouping information carried in the data summarizing instruction, and calculating the sum of the data of each group;

specifically, the grouping information carried by the data summarization instruction may be used as a basis for the node server to group the stored data. According to the grouping information, the node server divides a large amount of stored data into respective groups, and then calculates a sum of data of all data in each group, each sum corresponding to the summarized data of the corresponding group.

For example, the central server may determine original data to be processed according to the order of the order numbers, may send a data summarization instruction to the node servers, where the grouping information carried by the data summarization instruction may be the order number order, and may indicate the node servers to group the original data with the order numbers of 0-100, 101-200, 201-300 … into one group, respectively, for example, the length of each group may be changed according to the data amount. After dividing the stored data into a plurality of groups, the node server calculates the data sum of all the data in each group.

For example, the central server may determine original data to be processed according to a data value size order, may send a data summarization instruction to the node servers, where grouping information carried by the data summarization instruction may be the data value size order, and at this time, the node servers may group the stored original data according to the data value sizes, for example, divide data with data values of 0-5000, 5001 and 10000, 10001 and 15000, and … into one group, and of course, the length of each group may be changed according to the data amount.

S203, sending the data sum of each group to the central server;

specifically, the node servers transmit the data sums of the respective groups to a central server, which receives a plurality of data sums transmitted by a plurality of associated node servers.

S204, receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to data sum of a plurality of node servers;

after receiving the data sent by each node server, the central server can sort the data, then determines a critical value according to a data limit corresponding to a preset screening condition, and sends the critical value to the central server through a data bus.

For example, if the data sum is summarized by the node servers according to the order number, the central server may sort the data sum according to the order number, then select a data sum from the sorted data sum queue, and the sum obtained by adding the sorted data sum and the previous data sum is equal to the data quota, then the first order number of the selected data sum and the corresponding order number segment may be used as the critical value. For example, the data and corresponding order number is 601-700, then the order number 601 may be used as a threshold.

For example, if the data sums are grouped and summarized by the node servers according to the size of the data value, the central server may sort the data sums according to the order of the size of the data value, then select one data sum from the sorted data sum queue, the sum obtained by adding the sorted data sum and the previous data sum is equal to the data amount, and if the data value needs to be selected from large to small, the last data value of the selected data sum and the corresponding data value segment may be used as a critical value. For example, the data and corresponding data value are 10001-.

S205, identifying data to be processed in the data stored in the node server according to the critical value;

specifically, the node server screens the data to be processed from the stored data according to the data screening condition and the critical value.

For example, if the preference data value is large, the node server may identify data, of the stored data, whose data value is greater than the critical value as the data to be processed. If the order number is selected to be smaller, the node server may identify data, of the stored data, whose order number is smaller than the critical value as data to be processed.

And S206, processing the data to be processed according to a preset mode.

Specifically, after the node server identifies data to be processed from the stored data, the node server may process the data according to a preset processing mode. For example, the data stored by the node server is order data, and successful purchase applying information may be sent to the user corresponding to the data to be processed.

In this embodiment, data is stored in the node servers, and is not required to be completely transmitted to the central server, when data is screened, each node server firstly groups and summarizes data, the central server determines a critical value according to the summarized data, then each node server completes data processing according to the critical value, and the data processing process can be executed by each node server in parallel, so that the efficiency of data processing is improved.

Fig. 3 is a schematic flowchart of a data processing method according to a third embodiment of the present application, and as shown in fig. 3, the method includes:

s301, receiving a data summarizing instruction sent by a central server;

specifically, the execution subject of the present embodiment is a node server, and each node server stores data and can independently execute a task. When data in each node server needs to be processed uniformly, the central server sends a data summarizing instruction to each associated node server. And the node server receives the data summarizing instruction and performs subsequent processing according to the data summarizing instruction.

S302, grouping the data stored in the node server according to the grouping time periods, and calculating the sum of the data in each time period;

specifically, the data summarization instruction may carry a grouping time period, and the node server may divide the stored data into each group according to the grouping time period, and then calculate the data sum of each group, where the data sum is summarized data. For example, if the packet time period is 1 second, the node server divides data in the same second into one packet according to the generation time of the data, and then calculates the data sum of all data generated in each second. Of course, the length of the grouping time period can be adjusted according to actual requirements, for example, when the data distribution is more dispersed, the length of the grouping time period can be increased.

S303, receiving a critical value sent by the central server, where the critical value is determined by the central server according to summarized data of a plurality of node servers;

specifically, after receiving the sum of data sent by each node server, the central server may sort the received sum of data in a time sequence. For the arranged data sums, the first data sums are added one by one, whether the limiting condition of data screening is met or not is judged according to the adding result, when the sorted first data sums are added to a certain data sum a one by one, the obtained numerical value meets the preset condition, and the time period corresponding to the data sum a can be identified as a critical value required for screening the data to be processed.

And after determining the critical value, the central server transmits the critical value to each associated node server.

S304, determining a time point corresponding to the critical value;

specifically, the time point may be a start time of a time period in which the critical value corresponds to the data sum. For example, if the packet time period is 1 minute in length and the determined critical value corresponds to a time period of 12:00 to 12:01, 12:00:00 may be used as the time point corresponding to the critical value.

S305, identifying data of a time point corresponding to the data time information smaller than the critical value as the data to be processed in the data stored in the node server;

specifically, when data is screened, data of a fixed data limit needs to be screened according to a time sequence, the data stored by the node server includes time information of the data, and if a time point corresponding to the time information of a certain data is smaller than a time point corresponding to a critical value, the data can be identified as data to be processed.

S306, processing the data to be processed according to a preset mode;

specifically, the screened data to be processed is processed continuously according to a preset mode. If each data represents a subscription order, successful subscription information can be sent to the client corresponding to each order.

S307, acquiring target data processed according to the critical value, and reporting the target data to the central server;

specifically, when the central server determines the critical value, there may be a case where a first value obtained by adding the data before the critical value is smaller than a preset data amount, and the first value is added with the data corresponding to the critical value and is larger than the preset data amount. In this case, when the node server identifies the data to be processed according to the critical value, a small portion of the data to be processed may not be identified. The small portion of data to be processed is distributed among data corresponding to the threshold value and corresponding packets. The data in the group is target data, and each node server reports the target data to the central server.

S308, critical data returned by the central server aiming at the target data is received, and a target group to which the critical data belongs is determined;

after each node server reports the target data corresponding to the critical value and the data in the group to the central server, the central server sorts the target data according to different standards again, for example, the target data can be sorted according to the amount of money or the order number. And if one target data exists, the sum of a second numerical value obtained by adding the previous target data and the first numerical value is equal to a preset data limit, the data can be identified as critical data, and the critical data is sent to each node server.

Specifically, each node server receives critical data sent by the central server and determines a target group in which the critical data exists.

S309, processing the data in the target grouping according to a preset mode.

Specifically, if the data is selected to be sorted according to the size of the data value in step S308, the data screening criterion in the target packet is preferably that the data value is large, the data with the data value larger than the data value of the critical data in the target packet may be identified as the data to be processed, and the data to be processed is processed according to step S306.

In the embodiment, the central server does not need to screen a large amount of original data, and each node server commonly undertakes most of data screening work, so that the data processing efficiency is improved, and the calculation time is shortened; the central server only needs to receive a small amount of data sent by the node servers and then performs screening, so that the calculation work of the central server is reduced, and the storage space of the central server is saved.

Fig. 4 is a schematic flowchart of a data processing method according to a fourth embodiment of the present application, and as shown in fig. 4, the method includes:

s401, sending a data summarizing instruction to a plurality of node servers associated with the central server;

the execution main body of the embodiment is a central server, and the central server is a computing device capable of data processing and communication, and comprises a cloud server. The central server may be associated with a plurality of node servers, each node server capable of data transfer with the central server and independently performing tasks.

When data processing is required, data to be processed is selected from a large amount of data according to a specified rule. And the central server sends a data summarizing instruction to the associated node servers according to the specified rule, and instructs the node servers to summarize and report the data. After receiving the data summarization instruction sent by the central server, each node server summarizes the respective stored data according to the requirements of the central server, for example, calculates the data sum of each time period, and then sends the summarized data sum to the central server.

S402, receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping and summarizing stored data by each node server;

specifically, after receiving a data summarization instruction sent by the central server, each node server summarizes the stored data groups and sends the summarized data to the central server, and the central server receives and processes the summarized data sent by each node server.

S403, determining a critical value according to summarized data returned by the plurality of node servers;

specifically, the central server determines a critical value based on a preset data limit, and each sub-node selects data of the data to be processed according to the critical value and equals to the data limit. The central server sorts the received summarized data according to the grouping information carried by the data summarizing instruction, and adds the summarized data one by one in the sorted summarized data until the data value obtained by adding is equal to a preset data limit, and takes the information corresponding to the summarized data at the moment as a critical value.

For example, if the summary data is grouped and summarized by the node server according to the order number, the central server may sort the summary data according to order sequence, then select one summary data from the sorted summary data queue, and sum of summary data sorted before the summary data is equal to data amount, then the first order number of the order number segment corresponding to the selected summary data may be used as a critical value. For example, if the order number corresponding to the summary data is 601-700, the order number 601 may be used as a threshold.

For example, if the summarized data is grouped and summarized by the node servers according to the size of the data value, the central server may sort the summarized data according to the order of the size of the data value, then select one summarized data from the sorted summarized data queue, the sum of the summarized data sorted before the summarized data is equal to the data amount, and if the data value needs to be selected from large to small, the last data value of the data value segment corresponding to the selected summarized data may be used as the critical value. For example, the data value corresponding to the summarized data is 10001-.

S404, sending the critical value to the node servers to instruct the node servers to process the stored data according to the critical value.

Specifically, the central server issues the critical value to each node server, and each node server identifies data to be processed in the stored data according to the critical value and then processes the data to be processed according to a preset processing mode.

In the embodiment, the data is stored on each node server and does not need to be sent to the central server completely, so that the storage space of the central server is saved, and the requirement on the data transmission rate is reduced. The data screening is completed by the central server and the node servers together, and all the node servers can execute in parallel, so that the data processing efficiency is improved, and the data processing time is saved.

Fig. 5 is a schematic flowchart of a data processing method provided in the fifth embodiment of the present application, and as shown in fig. 5, the method includes:

s501, sending a data summarizing instruction to a plurality of node servers associated with the central server;

specifically, the central server sends a data summarization instruction to the associated node servers, where the data summarization instruction may carry a grouping time period, and each node server may divide the stored data into each group according to the grouping time period.

S502, receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping and summarizing stored data by each node server;

specifically, after receiving a data summarization instruction of the central server, each node server divides the stored data into groups according to time periods, calculates the sum of the data of each time period as summarized data, and sends the summarized data to the central server.

S503, sorting the summarized data returned by the node servers according to the grouping time period;

specifically, the central server sorts the received summary data in chronological order.

S504, the sorted summarized data are added one by one;

specifically, for the sorted summarized data, the respective summarized data may be added one by one. For example, sorting the summarized data to obtain a group of data a1, a2, a3, …, ax, …, an, and then adding the data one by one to calculate S1-a 1; s2 ═ S1+ a 2; …, respectively; sx is Sx-1+ ax; …, respectively; sn-1+ an.

S505, when the added numerical value meets a preset limiting condition, identifying a time period corresponding to the numerical value as a critical value of the data to be processed;

specifically, the predetermined limiting condition may include a predetermined data amount. And the sum of the data to be processed identified by each node server according to the critical value is equal to the data quota. The central server may sort the summarized data reported by each node server according to a time sequence, and for the sorted summarized data, if a sum obtained by adding all data before a certain summarized data is equal to a preset data limit, a time period corresponding to the summarized data may be used as a critical value. For example, if the predetermined data amount is B, S_x-1The time period corresponding to x may be a time threshold value.

S506, the critical value is sent to the node servers to instruct the node servers to process the stored data according to the critical value.

Specifically, the central server issues the determined time critical value to each node server, and each node server identifies data, which is generated at a time earlier than the time critical value, in the stored data as to-be-processed data, and then processes the to-be-processed data according to a preset processing mode.

In this embodiment, data is stored in the node servers, and is not required to be transmitted to the central server completely, when data is screened, each node server firstly groups and summarizes data, the central server determines a critical value according to the summarized data, and then each node server completes data screening according to the critical value and processes the screened data to be processed. The data processing can be executed in parallel through each node server, and the data processing efficiency is improved.

For better explanation, the data processing method in the present application will be described with a specific example.

Fig. 6 is a schematic diagram of a relationship between a central server and node servers according to a sixth embodiment of the present disclosure, where as shown in fig. 6, the central server is connected to a plurality of node servers through a data bus, and the communication bus includes middleware for message command transmission and data transmission. The central server comprises a headquarter service and a high-speed cache region, wherein the headquarter service comprises an executive program required by data processing, and the high-speed cache region can store data reported by each sub-node. The node server includes a node service including an execution program for data aggregation and processing and node data for storing data. When data processing is performed, a method shown in fig. 7 may be adopted, in which the central server issues a data summarization instruction to each node server through a data bus; after receiving the data summarizing instruction, each node server groups and summarizes data stored in the node data according to the data summarizing instruction and reports the summarized data to a central server through a data bus; the central server can store the received summarized data in a high-speed memory cache region, then call a program in the headquarter service to sort the summarized data reported by the node servers and determine a critical value, and issue the critical value to each node server through a data bus, and each node server then calls the program in the node service to screen and process the data according to the critical value issued by the headquarter.

Take data processing in the procurement process of security products as an example. The securities product procurement can be realized through a registration system (TA), the TA can distribute data to different node servers for processing on a central server according to a certain business rule, and the data storage adopts a database-dividing and table-dividing mode, so the TA has the characteristics of large data volume and data dispersion. For example, the TA service is used for illustration, for example, the product procurement limit amount is 100 billion, the sum of the orders received by each node server on the current day is 150 billion, at this time, 100 billion procurement orders need to be selected according to the time sequence, and if the time sequence is the same, the procurement orders with larger procurement amounts are preferred. During the data processing, the order can be selected as follows:

the central server issues a data summarizing instruction to each node server, instructs each node server to summarize the declaration amount in time groups and reports the total declaration amount in time groups;

after receiving the instruction of the central server, each node server collects the purchase order generated on each node server according to the transaction time, collects the total purchase sum of single time (granularity to second level), and sends the collected data to the central server.

The central server sorts the received grouped summarized data according to application time, confirms from the head group by group until 100 hundred million money is used up, finds an excess application time critical value, such as 14:01:58, and then sends the time critical value of 14:01:58 to each node server.

After receiving the application time critical value 14:01:58 sent by the central server, each node server parallelly applies for confirmation before the time of each node 14:01:58 is successful, and other procurement confirmation fails.

If a large number of procurement situations exist within one second (14:01:58) of the critical value and the time critical value cannot be located to a single procurement order, summarizing the procurement orders with the procurement time of 14:01:58 according to the procurement rules and the second priority condition of the magnitude of the procurement amount;

the central server sends an instruction to each node server to instruct each node to report a subscription order with subscription time of 14:01:58, determines a critical amount, for example, 5000, according to the reported subscription amount of each order, and then sends the critical amount 5000 to each node server. And (4) each node server confirms the success of the subscription order with the subscription amount larger than 5000 generated in the step (14:01:58) and completes the whole order subscription process.

Compared with the traditional method, the method for processing the data through the distributed data sorting and screening component can effectively reduce network and storage resources required by the same type of service scene, and can use each node server resource to process the data very efficiently.

For example: 10 node servers, the single purchase sum is 1 ten thousand, and the total collection sum is 100 hundred million. The total transaction number generated at present is 150 thousands, the actual application is 150 hundred million excess recruitment, and the application time is distributed in the time of 9:00: 00-15: 00: 00. If each node server collects the data according to the second, the grouping collection is performed according to 3600 seconds in 1 hour, 21600 grouped data are required to be transmitted in total, and 1500000 data are required to be transmitted in the traditional method, so the method reduces the burden of network transmission; the headquarter node only needs to sequence 21600 data stores, so that the storage use space is saved, and the calculation resources consumed by sequencing are reduced. The found critical value is distributed to each node server for processing, concurrent execution can be achieved, data results are directly stored after execution is completed, the central server does not need to be informed of each node server after screening is completed, service processing steps are reduced, and execution efficiency is improved.

In addition, the money subscription determination may be performed according to different rules. When selecting the amount subscription, different keywords can be selected, such as time, amount, order number, and the like; the user can directly select a certain keyword as a priority ranking basis, and then select another keyword as a second ranking basis. For example, the user may select the amount of money as the first sort criterion, and when the amounts of money are the same, the order number may be selected as the second sort criterion

Fig. 8 is a schematic structural diagram of a data processing apparatus according to a seventh embodiment of the present application, and as shown in fig. 8, the apparatus includes:

the instruction receiving module 81 is configured to receive a data summarizing instruction sent by the central server;

the data summarization module 82 is configured to group and summarize data stored in the node servers according to the data summarization instruction, and report summarized data to the central server;

a critical value receiving module 83, configured to receive a critical value sent by the central server, where the critical value is determined by the central server according to summarized data of multiple node servers;

and a data processing module 84, configured to process the data stored in the node server according to the critical value.

The data summarization module 82 includes the following sub-modules:

the calculation submodule is used for grouping the data stored in the node server according to the grouping information carried in the data summarizing instruction and calculating the sum of the data of each group;

a data and transmission submodule for transmitting the data sum of each packet to the central server.

The above calculation sub-module includes:

and each time period data and calculation unit is used for grouping the data stored in the node server according to the grouping time period and calculating the data sum in each time period.

The data processing module 84 includes the following sub-modules:

the identification submodule is used for identifying data to be processed in the data stored by the node server according to the critical value;

and the processing submodule is used for processing the data to be processed according to a preset mode.

In the above apparatus, the data stored in the node server has corresponding data time information, and the identification sub-module includes:

the determining unit is used for determining a time point corresponding to the critical value;

and the judging unit is used for identifying the data of the time point, corresponding to the critical value, of the data stored in the node server as the data to be processed.

The above-mentioned device still includes:

a target data reporting module, configured to obtain target data obtained according to the critical value processing, and report the target data to the central server;

the target grouping determination module is used for receiving critical data returned by the central server aiming at the target data and determining a target grouping to which the critical data belongs;

and the data processing module in the target grouping is used for processing the data in the target grouping according to a preset mode.

Fig. 9 is a schematic structural diagram of a data processing apparatus according to an eighth embodiment of the present application, and as shown in fig. 9, the apparatus includes:

an instruction sending module 91, configured to send a data summarization instruction to a plurality of node servers associated with the central server;

a data receiving module 92, configured to receive summarized data returned by the plurality of node servers according to the data summarizing instruction, where the summarized data is obtained by grouping and summarizing stored data by each node server;

a critical value determining module 93, configured to determine a critical value according to summarized data returned by the plurality of node servers;

a threshold sending module 94, configured to send the threshold to the plurality of node servers, so as to instruct the plurality of node servers to process the stored data according to the threshold.

In the above apparatus, the data summarizing instruction includes corresponding grouping information, the grouping information includes a grouping time period, and the critical value determining module 93 includes the following sub-modules:

the sorting submodule is used for sorting the summarized data returned by the node servers according to the grouping time period;

the calculation submodule one by one is used for adding the sorted summarized data one by one;

and the identification submodule is used for identifying the time period corresponding to the numerical value as a critical value of the data to be processed when the numerical value obtained by adding meets the preset limiting condition.

Fig. 10 is a schematic structural diagram of a server according to a ninth embodiment of the present application. As shown in fig. 10, the server 10 of this embodiment includes: at least one processor 1000 (only one shown in fig. 10), a memory 1001, and a computer program 1002 stored in the memory 1001 and executable on the at least one processor 1000, the processor 1000 implementing the steps of any of the various method embodiments described above when executing the computer program 1002.

The server 10 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The server may include, but is not limited to, a processor 1000, a memory 1001. Those skilled in the art will appreciate that fig. 10 is merely an example of a server 10 and does not constitute a limitation on server 10, and may include more or fewer components than shown, or some components in combination, or different components, such as input output devices, network access devices, etc.

The processor 1000 may be a Central Processing Unit (CPU), and the processor 1000 may also be other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The storage 1001 may be an internal storage unit of the server 10 in some embodiments, for example, a hard disk or a memory of the server 10. In other embodiments, the memory 1001 may also be an external storage device of the server 10, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card (FlashCard), or the like, provided on the server 10. Further, the memory 1001 may also include both an internal storage unit and an external storage device of the server 10. The memory 1001 is used for storing an operating system, an application program, a BootLoader (BootLoader), data, and other programs, such as program codes of the computer program. The memory 1001 may also be used to temporarily store data that has been output or is to be output.

It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.

The embodiments of the present application provide a computer program product, which when running on a server, enables the server to implement the steps in the above method embodiments when executed.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/server, a recording medium, computer memory, Read-only memory (ROM), random-access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A data processing method is applied to a node server, and the method comprises the following steps:

receiving a data summarizing instruction sent by a central server;

2. The method of claim 1, wherein grouping and summarizing the data stored by the node servers according to the data summarizing instructions and reporting the summarized data to the central server comprises:

grouping the data stored by the node server according to the grouping information carried in the data summarizing instruction, and calculating the sum of the data of each group;

sending the data sum of each packet to the central server.

3. The method according to claim 2, wherein the grouping information includes a grouping time period, and the grouping the data stored in the node server according to the grouping information carried in the data summarization instruction, and calculating the data sum of each group includes:

and grouping the data stored by the node server according to the grouping time periods, and calculating the data sum in each time period.

4. The method of any of claims 1-3, wherein the processing the data stored by the node server according to the critical value comprises:

identifying data to be processed in the data stored by the node server according to the critical value;

and processing the data to be processed according to a preset mode.

5. The method of claim 4, wherein the data stored in the node servers respectively have corresponding data time information, and the identifying the data to be processed in the data stored in the node servers according to the critical value comprises:

determining a time point corresponding to the critical value;

and identifying data at a time point corresponding to the critical value, in which the data time information is smaller than the critical value, as the data to be processed, in the data stored in the node server.

6. The method of claim 1, 2, 3, or 5, further comprising:

acquiring target data processed according to the critical value, and reporting the target data to the central server;

critical data returned by the central server aiming at the target data is received, and a target group to which the critical data belongs is determined;

and processing the data in the target packet according to a preset mode.

7. A data processing method applied to a central server, the method comprising:

8. The method of claim 7, wherein the data summarization instructions comprise respective grouping information, the grouping information comprising a grouping time period, and wherein determining a threshold from summarized data returned by the plurality of node servers comprises:

sorting summarized data returned by the node servers according to the grouping time period;

adding the sorted summarized data one by one;

and when the added numerical value meets a preset limiting condition, identifying a time period corresponding to the numerical value as a critical value of the data to be processed.

9. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 8.