CN111259045B - Data processing method, device, server and medium - Google Patents
Data processing method, device, server and medium Download PDFInfo
- Publication number
- CN111259045B CN111259045B CN202010054694.2A CN202010054694A CN111259045B CN 111259045 B CN111259045 B CN 111259045B CN 202010054694 A CN202010054694 A CN 202010054694A CN 111259045 B CN111259045 B CN 111259045B
- Authority
- CN
- China
- Prior art keywords
- data
- server
- node
- critical value
- central server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 69
- 238000000034 method Methods 0.000 claims abstract description 65
- 238000003860 storage Methods 0.000 claims abstract description 17
- 230000008569 process Effects 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 21
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 238000012216 screening Methods 0.000 description 17
- 230000005540 biological transmission Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000004044 response Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000012790 confirmation Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007115 recruitment Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Abstract
The application is applicable to the technical field of data processing, and provides a data processing method, a data processing device, a server and a medium, wherein the method comprises the following steps: receiving a data summarizing instruction sent by a central server; grouping and summarizing the data stored by the node server according to the data summarizing instruction, and reporting summarized data to the central server; receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers; and processing the data stored by the node server according to the critical value. By the method, the data processing efficiency can be improved, and the storage space is saved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method, a data processing device, a server, and a medium.
Background
In the field of computer technology, it is often necessary to process large amounts of data. Such as data collection, data sorting, and data screening. When the amount of data to be processed is large, the whole data processing process consumes much resources and takes a long time.
Distributed computing can split a large problem into multiple sub-problems and then solve the sub-problems at individual split nodes, thereby shortening the time to solve the entire large problem. However, at present, although distributed computing can be used to collect data, when data sorting and screening are performed, a large amount of data is concentrated to be processed, all data is sequentially processed by adopting the same computing device, a large amount of network transmission and storage resources are required to be consumed, and the computing time is relatively long.
Disclosure of Invention
The embodiment of the application provides a data processing method, a device, a server and a medium, which can solve the problems of more data processing consumption resources and low efficiency.
In a first aspect, an embodiment of the present application provides a data processing method, applied to a node server, where the method includes:
receiving a data summarizing instruction sent by a central server;
grouping and summarizing the data stored by the node server according to the data summarizing instruction, and reporting summarized data to the central server;
receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers;
And processing the data stored by the node server according to the critical value.
In a second aspect, an embodiment of the present application provides a data processing method, applied to a central server, where the method includes:
transmitting a data summarization instruction to a plurality of node servers associated with the central server;
receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping and summarizing stored data by each node server;
determining a critical value according to summarized data returned by the plurality of node servers;
and sending the critical value to the plurality of node servers to instruct the plurality of node servers to process the stored data according to the critical value.
In a third aspect, an embodiment of the present application provides a data processing apparatus, applied to a node server, the apparatus including:
the instruction receiving module is used for receiving a data summarizing instruction sent by the central server;
the data summarizing module is used for summarizing the data stored by the node server in groups according to the data summarizing instruction and reporting the summarized data to the central server;
The critical value receiving module is used for receiving a critical value sent by the central server, and the critical value is determined by the central server according to summarized data of a plurality of node servers;
and the data processing module is used for processing the data stored by the node server according to the critical value.
In a fourth aspect, an embodiment of the present application provides a data processing apparatus, applied to a central server, including:
the instruction sending module is used for sending data summarizing instructions to a plurality of node servers associated with the central server;
the data receiving module is used for receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, and each node server carries out grouping summarization on the stored data to obtain the summarized data;
the critical value determining module is used for determining critical values according to summarized data returned by the plurality of node servers;
and the critical value sending module is used for sending the critical value to the plurality of node servers so as to instruct the plurality of node servers to process the stored data according to the critical value.
In a fifth aspect, an embodiment of the present application provides a server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the method according to the first or second aspect when executing the computer program.
In a sixth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which, when executed by a processor, implements a method as described in the first or second aspect above.
In a seventh aspect, embodiments of the present application provide a computer program product which, when run on a server, causes the server to perform the method of any one of the first or second aspects above.
Compared with the prior art, the embodiment of the application has the beneficial effects that: in the embodiment of the application, a central server is associated with a plurality of node servers, each node server receives a data summarizing instruction sent by the central server, then data stored by the node servers are summarized in groups according to the data summarizing instruction, and summarized data are reported to the central server; the central server determines a critical value according to summarized data of a plurality of node servers, the node servers receive the critical value sent by the central server, and then the data stored by the node servers are processed according to the critical value. In the data processing process, all original data are not required to be sent to the central server, but the node server is used for summarizing the data and then sending the summarized data to the central server, so that the data quantity required to be transmitted is reduced, and the storage space of the central server is saved. According to the embodiment, the node server and the central server perform data processing together, so that the data processing efficiency is improved, and the calculation time is shortened.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data processing method according to a first embodiment of the present application;
fig. 2 is a flow chart of a data processing method according to a second embodiment of the present application;
fig. 3 is a flow chart of a data processing method according to a third embodiment of the present application;
fig. 4 is a flow chart of a data processing method according to a fourth embodiment of the present application;
fig. 5 is a flow chart of a data processing method according to a fifth embodiment of the present application;
fig. 6 is a schematic diagram of a relationship between a central server and a node server according to a sixth embodiment of the present application;
fig. 7 is a flow chart of a data processing method according to a sixth embodiment of the present application;
FIG. 8 is a schematic diagram of a data processing apparatus according to a seventh embodiment of the present application;
FIG. 9 is a schematic diagram of a data processing apparatus according to an eighth embodiment of the present application;
fig. 10 is a schematic structural diagram of a server according to a ninth embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in the present description and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
Furthermore, the terms "first," "second," "third," and the like in the description of the present specification and in the appended claims, are used for distinguishing between descriptions and not necessarily for indicating or implying a relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
Fig. 1 is a flow chart of a data processing method according to a first embodiment of the present application, as shown in fig. 1, where the method includes:
s101, receiving a data summarizing instruction sent by a central server;
in the embodiment of the application, the central server is associated with a plurality of node servers, communication can be carried out between the central server and the node servers, and each node server can independently execute tasks. The execution body of the embodiment is a node server, and the node server is a server capable of performing data screening and data transmission, including a cloud server.
Specifically, when data processing is required, the central server sends a data summarizing instruction to the node server, the node server receives the data summarizing instruction sent by the central server, and subsequent data processing is performed according to the data summarizing instruction.
S102, grouping and summarizing the data stored by the node server according to the data summarizing instruction, and reporting summarized data to the central server;
specifically, a large amount of raw data is stored in the node server, and the raw data may include data generation time, data value, order number, client information, and the like. And the node server groups the stored original data according to a certain characteristic according to the data summarizing instruction. For example, the packets may be grouped by time period, order number, or data value. And adding the data values corresponding to all the data of each group to obtain summarized data corresponding to each group, and sending the summarized data of each group to a central server. And after receiving the data summarizing instruction, the plurality of node servers associated with the central server all send summarized data to the central server.
S103, receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers;
the threshold may be a criterion for data screening. For example, the threshold may be a data value, and if the data value of the original data is greater than the threshold, the original data is selected to be processed; the threshold may be a time value, and if the generation time of the original data is earlier than the threshold, the original data is selected to be processed data; the threshold may be an order number, and if the order number of the original data is before the threshold, the original data is selected to be processed data.
Specifically, after receiving the summarized data sent by each node server, the central server sorts the summarized data, then determines a critical value, and sends the critical value to each node server through a data bus, and the node server receives the critical value.
S104, processing the data stored by the node server according to the critical value.
Each node server can independently execute tasks, so that a large amount of original data can be generated in the process of processing the tasks, and the data can be processed according to the needs. Illustratively, each node server receives the user's order, and then needs to confirm the eligible order according to the rule specified by the central server, and sends the purchase success information to the user corresponding to the eligible order.
Specifically, the node server screens out the data required in the stored original data according to the critical value determined by the central server, and then processes the data according to a preset processing mode. If the data is a user purchase order, a plurality of user purchase orders are generated on each node server, and the total amount of all the purchase orders is greater than the preset total amount, the data is required to be screened, the user who is successful in purchase is determined, and then the user purchase order which is successful in purchase is processed. In the data screening, the purchase order may be selected according to different conditions, and for example, an order priority with a large amount may be selected. The central server selects the critical value as an amount value, and the node server can select the orders with the amount greater than or equal to the critical value from the stored orders after receiving the critical value, and then sends the purchase success information to the users corresponding to the orders.
In this embodiment, the data are stored on each node server, and all the data do not need to be sent to the servers, but the node servers summarize the data and send the summarized data to the central server, so that the storage space of the central server is saved, and the pressure of data transmission is reduced. The screening of the data is completed by the central server and the node servers, and each node server can execute in parallel, so that the data processing efficiency is improved, and the data processing time is saved.
Fig. 2 is a flow chart of a data processing method according to a second embodiment of the present application, as shown in fig. 2, where the method includes:
s201, receiving a data summarizing instruction sent by a central server;
the implementation main body of the embodiment is a node server, the node server is associated with a central server, and the data summarizing instruction is sent to the node server by the central server and can carry grouping information.
S202, grouping the data stored by the node server according to grouping information carried in the data summarizing instruction, and calculating the data sum of each grouping;
specifically, the grouping information carried by the data summarization instruction can be used as the basis of the node server to the stored data grouping. Based on the grouping information, the node server divides the stored large amount of data into individual groups, and then calculates the data sum of all the data in each group, each data sum corresponding to the summarized data of the corresponding group.
For example, the central server may determine the original data to be processed according to the order number sequence, may send a data summarizing instruction to the node server, and the packet information carried by the data summarizing instruction may be the order number sequence, for example, may instruct the node server to group the original data with order numbers of 0-100, 101-200, 201-300 and … in units of 100, and divide the original data with order numbers into a group respectively, where the length of each group may vary according to the data amount. After the node server divides the stored data into a plurality of subgroups, the data sum of all the data in each subgroup is calculated.
For example, the central server may determine the original data to be processed according to the data value size sequence, and may send a data summarizing instruction to the node server, where the packet information carried by the data summarizing instruction may be the data value size sequence, and at this time, the node server may group the stored original data according to the data value size, for example, divide the data values in the range of 0-5000, 5001-10000, 10001-15000, … into a group respectively, and of course, the length of each packet may vary according to the data amount.
S203, data sum of each packet is sent to the central server;
specifically, the node servers send the data sums of the respective packets to a central server that receives the plurality of data sums sent by the plurality of associated node servers.
S204, receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to the data sum of a plurality of node servers;
after receiving the data sent by each node server and the data, the central server can sort the data, then determine a critical value according to the data limit corresponding to the preset screening condition, and send the critical value to the central server through a data bus.
For example, if the data sums are grouped and summarized by the node server according to the order numbers, the central server may sort the data sums according to the order number sequence, then select a data sum from the sorted data sums and the queue, and the sum obtained by adding the data sum before the data sum is sorted, then the first order number of the selected data sum corresponding to the order number fragment may be used as the critical value. For example, the data and corresponding order number 601-700, then order number 601 may be used as the threshold.
For example, if the data sums are grouped and summarized by the node server according to the size of the data values, the central server may sort the data sums according to the size order of the data values, then select one data sum from the sorted data sums and the queue, and the sum obtained by adding the data sums before the data sum is equal to the data quota, and if the data values need to be selected from large to small, the last data value of the selected data sum corresponding to the data value segment may be used as the critical value. For example, the data and the corresponding data value are 10001-20000, the data value 20000 can be used as a threshold.
S205, identifying data to be processed in the data stored by the node server according to the critical value;
specifically, according to the data screening conditions and the critical value, the node server screens the data to be processed from the stored data.
For example, if the priority data value is relatively large, the node server may identify data having a data value greater than the threshold value as data to be processed among the stored data. If the priority order number is smaller, the node server can identify the data with the order number smaller than the critical value in the stored data as the data to be processed.
S206, processing the data to be processed according to a preset mode.
Specifically, after the node server identifies the data to be processed from the stored data, the data may be processed according to a preset processing mode. For example, the data stored in the node server is order data, and the purchase success information can be sent to the user corresponding to the data to be processed.
In this embodiment, the data are stored in the node servers, all the data are not required to be transmitted to the central server, when the data are screened, each node server firstly performs grouping summarization on the data, the central server determines a critical value according to the summarized data, and then each node server completes data processing according to the critical value, and the data processing process can be executed in parallel by each node server, so that the data processing efficiency is improved.
Fig. 3 is a flow chart of a data processing method according to a third embodiment of the present application, as shown in fig. 3, where the method includes:
s301, receiving a data summarizing instruction sent by a central server;
specifically, the execution body of the embodiment is a node server, and each node server stores data and can independently execute tasks. When the data in each node server needs to be processed uniformly, the central server sends a data summarizing instruction to each associated node server. And the node server receives the data summarizing instruction and performs subsequent processing according to the data summarizing instruction.
S302, grouping the data stored by the node server according to the grouping time periods, and calculating the data sum in each time period;
specifically, the data summarizing instruction may carry a grouping time period, and the node server may divide the stored data into each group according to the grouping time period, and then calculate a data sum of each group, where the data sum is summarized data. For example, if the packet period is 1 second, the node server will divide the data within the same second into one packet according to the generation time of the data, and then calculate the data sum of all the data generated within each second. Of course, the length of the grouping time period can be adjusted according to actual requirements, for example, when the data distribution is relatively scattered, the length of the grouping time period can be increased.
S303, receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to summarized data of a plurality of node servers;
specifically, after receiving the data sums sent by the node servers, the central server may sort the received data sums in time sequence. And for the arranged data sums, adding the first data sums one by one, judging whether the limiting condition of data screening is met according to the adding result, and when the obtained numerical value meets the preset condition when the first data sums are added to a certain data sum a one by one, identifying the time period corresponding to the data sum a as a critical value required for screening the data to be processed.
After the central server determines the threshold, the central server issues the threshold to each associated node server.
S304, determining a time point corresponding to the critical value;
specifically, the above-described time point may be a start time of a period of time for which the critical value corresponds to the data sum. For example, the grouping time period is 1 minute, and the determined time period corresponding to the critical value is 12:00-12:01, and then 12:00:00 can be taken as the time point corresponding to the critical value.
S305, identifying the data of the time point, corresponding to the data time information being smaller than the critical value, as the data to be processed from the data stored in the node server;
specifically, when screening data, the data with fixed data unit need to be screened according to time sequence, the data stored by the node server includes time information of the data, and if the time point corresponding to the time information of a certain data is smaller than the time point corresponding to the critical value, the data can be identified as the data to be processed.
S306, processing the data to be processed according to a preset mode;
specifically, the screened data to be processed is continuously processed according to a preset mode. If each data represents a purchase order, the purchase success information may be sent to the customer corresponding to each order.
S307, obtaining target data obtained through processing according to the critical value, and reporting the target data to the central server;
specifically, when the central server determines the critical value, there may be a case that the data before the critical value and the first value obtained by adding are smaller than the preset data amount, and the first value plus the data corresponding to the critical value and the data larger than the preset data amount. In this case, when the node server identifies the data to be processed according to the threshold value, a small portion of the data to be processed cannot be identified. The small portion of the data to be processed is distributed among the data and the corresponding packets corresponding to the threshold. The data in the packet is target data, and each node server reports the target data to the central server.
S308, receiving critical data returned by the central server aiming at the target data, and determining target packets to which the critical data belong;
after each node server reports the target data corresponding to the critical value and the data in the group to the central server, the central server ranks the target data according to different standards again, for example, the target data can be ranked according to the amount of money or the order number. And if one target data exists, the sum of the obtained second value and the first value of the previous target data is equal to a preset data limit, the data can be identified as critical data, and the critical data is issued to each node server.
Specifically, each node server receives critical data sent by the central server, and determines a target packet in which the critical data exists.
S309, processing the data in the target packet according to a preset mode.
Specifically, if the sorting of the data according to the data value is selected in step S308, the data screening criteria of the data in the target packet is a priority of large data value, and the data with the data value greater than the data value of the critical data in the target packet may be identified as the data to be processed, and the data to be processed is processed according to the step S306.
In the embodiment, the central server does not need to screen a large amount of original data, and all node servers share most data screening work, so that the data processing efficiency is improved, and the calculation time is shortened; the central server only needs to receive a small amount of data sums sent by the node servers and then screen, so that the calculation work of the central server is reduced, and the storage space of the central server is saved.
Fig. 4 is a flow chart of a data processing method according to a fourth embodiment of the present application, as shown in fig. 4, where the method includes:
s401, sending a data summarizing instruction to a plurality of node servers associated with the central server;
the execution subject of the embodiment is a central server, which is a computing device capable of data processing and communication, including a cloud server. The central server may be associated with a plurality of node servers, each capable of data transmission with the central server and capable of independently performing tasks.
When data processing is needed, firstly, selecting data to be processed from a large amount of data according to a specified rule. And the central server sends a data summarizing instruction to the associated plurality of node servers according to the specified rule, and instructs each node server to summarize and report the data. After receiving the data summarizing instruction sent by the central server, each node server summarizes the data stored by each node server according to the requirement of the central server, for example, calculates the data sum of each time period, and then sends the summarized data sum to the central server.
S402, receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping summarization of stored data by each node server;
specifically, after receiving the data summarizing instruction sent by the central server, each node server can summarize the stored data packets and send summarized data to the central server, and the central server processes the summarized data sent by each node server.
S403, determining a critical value according to summarized data returned by the plurality of node servers;
specifically, the central server determines a threshold value based on a preset data unit, and each of the partial nodes selects data of the data to be processed according to the threshold value and the data unit is equal to the data unit. And the central server sorts the received summarized data according to grouping information carried by the data summarizing instruction, adds the summarized data one by one in the sorted summarized data until the added data value is equal to a preset data limit, and takes the information corresponding to the summarized data at the moment as a critical value.
For example, if the summary data are grouped and summarized by the node server according to the order numbers, the central server may sort the summary data according to the order sequence, then select one summary data from the sorted summary data queue, and sum the summary data sorted before the summary data to be equal to the data unit, then the first order number of the order number segment corresponding to the selected summary data may be used as the critical value. For example, if the order number corresponding to the summary data is 601-700, the order number 601 may be used as the threshold value.
For example, if the summary data are grouped and summarized by the node server according to the size of the data values, the central server may sort the summary data according to the size sequence of the data values, then select one summary data from the sorted summary data queue, and sum the summary data sorted before the summary data to be equal to the data amount, if the data values need to be selected from large to small, then use the last data value of the data value segment corresponding to the selected summary data as the critical value. For example, when the data value corresponding to the summary data is 10001 to 20000, the data value 20000 may be used as a threshold value.
And S404, the critical value is sent to the plurality of node servers to instruct the plurality of node servers to process the stored data according to the critical value.
Specifically, the central server issues the critical value to each node server, and each node server identifies data to be processed in the stored data according to the critical value and then processes the data to be processed according to a preset processing mode.
In this embodiment, the data is stored on each node server, and is not required to be all sent to the central server, so that the storage space of the central server is saved, and the requirement on the data transmission rate is reduced. The screening of the data is completed by the central server and the node servers, and each node server can execute in parallel, so that the data processing efficiency is improved, and the data processing time is saved.
Fig. 5 is a flow chart of a data processing method provided in a fifth embodiment of the present application, as shown in fig. 5, where the method includes:
s501, sending a data summarizing instruction to a plurality of node servers associated with the central server;
specifically, the central server sends data summarization instructions to the associated plurality of node servers, the data summarization instructions may carry grouping time periods, and each node server may divide the stored data into each subgroup according to the grouping time periods.
S502, receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping summarization of stored data by each node server;
specifically, after each node server receives the data summarizing instruction of the central server, the stored data are divided into groups according to time periods, then the data sum of each time period is calculated and used as summarizing data, and the summarizing data are sent to the central server.
S503, sorting summarized data returned by the plurality of node servers according to the grouping time period;
specifically, the central server sorts the received summary data in time order.
S504, adding the sorted summarized data one by one;
specifically, for the sorted summarized data, the respective summarized data may be added one by one. For example, sorting the summary data to obtain a set of data a1, a2, a3, …, ax, …, an, and then adding one by one to calculate s1=a1; s2=s1+a2; …; sx=sx-1+ax; …; sn=sn-1+an.
S505, when the added numerical value meets a preset limiting condition, identifying a time period corresponding to the numerical value as a critical value of data to be processed;
specifically, the preset limiting condition may include a preset data amount. And the sum of the data to be processed identified by each node server according to the critical value is equal to the data limit. The central server may sort the summary data reported by each node server according to a time sequence, and for the sorted summary data, if the sum obtained by adding all the data before a certain summary data is equal to a preset data unit, the time period corresponding to the summary data may be used as a critical value. For example, the preset data limit is B, if S x-1 The time period corresponding to x may be a time critical value.
And S506, the critical value is sent to the plurality of node servers to instruct the plurality of node servers to process the stored data according to the critical value.
Specifically, the central server transmits the determined time critical value to each node server, and each node server identifies the data which is generated in the stored data and is earlier than the time critical value as the data to be processed, and then processes the data to be processed according to a preset processing mode.
In this embodiment, the data are stored in the node servers, and are not required to be all transmitted to the central server, and when the data are screened, each node server firstly performs grouping summarization on the data, then the central server determines a critical value according to the summarized data, and then each node server finishes data screening according to the critical value and processes the screened data to be processed. The data processing can be executed in parallel through each node server, so that the data processing efficiency is improved.
For better explanation, the data processing method in the present application will be described with a specific example.
Fig. 6 is a schematic diagram of a relationship between a central server and node servers according to a sixth embodiment of the present application, where, as shown in fig. 6, the central server is connected to a plurality of node servers through a data bus, and the communication bus includes middleware for message command transmission and data transmission. The central server comprises a headquarter service and a cache region, the headquarter service comprises an execution program required by data processing, and the cache region of the cache memory can store data reported by each sub-node. The node server includes a node service including an execution program of the data aggregation processing and node data for storing data. When data processing is performed, a method shown in fig. 7 can be adopted, and a central server issues a data summarizing instruction to each node server through a data bus; after each node server receives the data summarizing instruction, data stored in the node data are summarized in groups according to the data summarizing instruction, and summarized data are reported to the central server through a data bus; the central server can store the received summarized data in the cache area of the high-speed memory, then call the program in the headquarter service to order the summarized data reported by the node servers, determine the critical value, send the critical value to each node server through the data bus, and call the program in the node service to screen and process the data according to the critical value sent by the headquarter.
Take data processing during the process of securities product purchase as an example. The security product purchase can be realized through a registration system (TransferAgent, TA), and the TA can distribute data to different node servers for processing according to certain business rules on a central server, and the data storage adopts a database and table dividing mode, so that the TA has the characteristics of large data volume and scattered data. For example, the TA service is used to illustrate that the limited amount of product purchase is 100 hundred million, the total amount of orders received by each node server on the same day is 150 hundred million, and when the total amount of the orders received by each node server is 150 hundred million, 100 hundred million purchase orders need to be selected according to the time sequence, and if the time sequence is the same, the purchase orders with larger purchase amount are prioritized. In the process of data processing, the selection of the order can be performed according to the following method:
the central server issues a data summarizing instruction to each node server, instructs each node server to summarize the declaration amount according to time groups and reports the total amount of the declaration amount according to time groups;
after receiving the instruction of the central server, each node server gathers the purchase orders generated on each node server according to the transaction time, gathers the total number of the purchase amounts in a single time (granularity to second level), and reports the total number of the purchases to the central server.
The central server sorts the received packet summary data according to the application time, confirms from the head part group by group until 100 hundred million amounts are full, finds out an excessive application time threshold, for example, 14:01:58, and then issues the time threshold of 14:01:58 to each node server.
After receiving the application time critical value 14:01:58 sent by the central server, each node server applies for confirmation successfully before the time of each node 14:01:58 in parallel, and the rest of the application confirmation fails.
If a large number of buying cases exist in the critical value for one second (14:01:58), and the time critical value cannot be positioned to a single buying order, according to the buying rule, the second priority condition is the buying amount, and the buying orders with the buying time of 14:01:58 are summarized according to the amount of money;
the central server sends instructions to each node server to instruct each node to report the purchase orders with the subscription time of 14:01:58, determines the critical amount, such as 5000, according to the reported purchase amount of each order, and then issues the critical amount 5000 to each node server. And each node server confirms the purchase order with the purchase amount greater than 5000 generated in the 14:01:58 to be successful, and completes the whole order purchase process.
Compared with the traditional method, the method for processing the data through the distributed data sorting and screening component can effectively reduce network and storage resources required by similar service scenes, and can use the resources of each node server to efficiently process the data.
For example: and 10 node servers, wherein the single purchase amount is 1 ten thousand, and the total recruitment amount is 100 hundred million. The total transaction number generated is 150 ten thousand, the actual application is overage recruitment of 150 hundred million, and the application time is distributed in the early 9:00:00-15:00:00 time. If each node server gathers according to seconds, grouping gathers according to 1 hour 3600 seconds, and a total of 21600 pieces of grouping data need to be transmitted, and 1500000 pieces of data need to be transmitted in the traditional method, so that the method reduces the network transmission burden; the headquarter node only needs to sort 21600 data stores, so that the storage and use space is saved, and the calculation resources consumed by sorting are reduced. The critical value is found and issued to each node server for distributed processing, the distributed processing can be executed concurrently, the data result is directly stored after the execution is completed, the central server is not required to be informed of each node server after the screening is completed, the service processing steps are reduced, and the execution efficiency is improved.
In addition, it is also possible to make monetary purchase determinations according to different rules. In the case of selecting the amount subscription, different keywords, such as time, amount, order number, etc., may be selected; the user can directly select one keyword as the priority ranking basis and select the other keyword as the second order ranking basis. For example, the user may select the amount as the first ranking basis, and when the amounts are the same, the order number may be selected as the second ranking basis
Fig. 8 is a schematic structural diagram of a data processing apparatus according to a seventh embodiment of the present application, as shown in fig. 8, where the apparatus includes:
the instruction receiving module 81 is configured to receive a data summarizing instruction sent by the central server;
the data summarizing module 82 is configured to group and summarize data stored in the node server according to the data summarizing instruction, and report summarized data to the central server;
a threshold receiving module 83, configured to receive a threshold sent by the central server, where the threshold is determined by the central server according to summarized data of a plurality of node servers;
and the data processing module 84 is configured to process the data stored in the node server according to the critical value.
The data summarization module 82 includes the following sub-modules:
the calculation sub-module is used for grouping the data stored by the node server according to the grouping information carried in the data summarizing instruction, and calculating the data sum of each grouping;
and the data and sending sub-module is used for sending the data sum of each packet to the central server.
The calculation submodule includes:
and each time period data and calculating unit is used for grouping the data stored by the node server according to the grouping time period and calculating the data sum in each time period.
The data processing module 84 includes the following sub-modules:
the identification sub-module is used for identifying data to be processed in the data stored by the node server according to the critical value;
and the processing sub-module is used for processing the data to be processed according to a preset mode.
In the above device, the data stored in the node server respectively has corresponding data time information, and the identification sub-module includes:
a determining unit, configured to determine a time point corresponding to the critical value;
and the judging unit is used for identifying the data of the time point corresponding to which the data time information is smaller than the critical value as the data to be processed in the data stored in the node server.
The device further comprises:
the target data reporting module is used for acquiring target data obtained through processing according to the critical value and reporting the target data to the central server;
the target packet determining module is used for receiving critical data returned by the central server aiming at the target data and determining target packets to which the critical data belong;
and the data processing module in the target packet is used for processing the data in the target packet according to a preset mode.
Fig. 9 is a schematic structural diagram of a data processing apparatus according to an eighth embodiment of the present application, as shown in fig. 9, where the apparatus includes:
an instruction sending module 91, configured to send a data summarizing instruction to a plurality of node servers associated with the central server;
the data receiving module 92 is configured to receive summarized data returned by the plurality of node servers according to the data summarizing instruction, where the summarized data is obtained by grouping and summarizing the stored data by each node server;
a critical value determining module 93, configured to determine a critical value according to summarized data returned by the plurality of node servers;
and a critical value sending module 94, configured to send the critical value to the plurality of node servers, so as to instruct the plurality of node servers to process the stored data according to the critical value.
In the above apparatus, the data summarizing instruction includes corresponding grouping information, where the grouping information includes a grouping time period, and the threshold determining module 93 includes the following sub-modules:
the sequencing sub-module is used for sequencing the summarized data returned by the plurality of node servers according to the grouping time period;
the one-by-one computing sub-module is used for adding the ordered summarized data one by one;
and the identification sub-module is used for identifying the time period corresponding to the value as the critical value of the data to be processed when the added value meets the preset limiting condition.
Fig. 10 is a schematic structural diagram of a server according to a ninth embodiment of the present application. As shown in fig. 10, the server 10 of this embodiment includes: at least one processor 1000 (only one shown in fig. 10), a memory 1001 and a computer program 1002 stored in the memory 1001 and executable on the at least one processor 1000, the processor 1000 implementing the steps in any of the various method embodiments described above when executing the computer program 1002.
The server 10 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The server may include, but is not limited to, a processor 1000, a memory 1001. It will be appreciated by those skilled in the art that fig. 10 is merely an example of server 10 and is not intended to limit server 10, and may include more or fewer components than shown, or may combine certain components, or different components, such as may also include input-output devices, network access devices, etc.
The processor 1000 may be a central processing unit (CentralProcessingUnit, CPU), and the processor 1000 may also be other general purpose processors, digital signal processors (DigitalSignalProcessor, DSP), application specific integrated circuits (ApplicationSpecificIntegratedCircuit, ASIC), off-the-shelf programmable gate arrays (Field-ProgrammableGateArray, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 1001 may in some embodiments be an internal storage unit of the server 10, such as a hard disk or a memory of the server 10. The memory 1001 may also be an external storage device of the server 10 in other embodiments, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. provided on the server 10. Further, the memory 1001 may further include both an internal storage unit and an external storage device of the server 10. The memory 1001 is used for storing an operating system, an application program, a boot loader (BootLoader), data, and other programs, etc., such as program codes of the computer program. The memory 1001 may also be used to temporarily store data that has been output or is to be output.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps for implementing the various method embodiments described above.
Embodiments of the present application provide a computer program product which, when run on a server, causes the server to perform steps that enable the implementation of the method embodiments described above.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/server, recording medium, computer memory, read-only memory (ROM), random access memory (RAM, randomAccessMemory), electrical carrier signal, telecommunication signal, and software distribution medium. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference may be made to related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. For example, the apparatus/network device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (10)
1. A data processing method, applied to a node server, the method comprising:
receiving a data summarizing instruction sent by a central server;
grouping and summarizing data stored by a node server according to the data summarizing instruction, and reporting summarized data to the central server, wherein the summarized data comprises data of each grouping and data of each group;
Receiving a critical value sent by the central server, wherein the critical value is determined by the central server according to a preset data limit and a sequencing result of summarized data of a plurality of node servers, and the node servers select data of data to be processed and equal to the data limit according to the critical value;
and processing the data stored by the node server according to the critical value.
2. The method of claim 1, wherein grouping and summarizing data stored by a node server according to the data summarization instruction and reporting summarized data to the central server comprises:
grouping the data stored by the node server according to the grouping information carried in the data summarizing instruction, and calculating the data sum of each grouping;
the data sum of each packet is sent to the central server.
3. The method of claim 2, wherein the grouping information includes a grouping time period, the grouping the data stored by the node server according to the grouping information carried in the data summarizing instruction, and calculating a data sum of each grouping includes:
and grouping the data stored by the node server according to the grouping time periods, and calculating the data sum in each time period.
4. A method according to any of claims 1-3, wherein said processing data stored by said node server according to said threshold comprises:
identifying data to be processed in the data stored by the node server according to the critical value;
and processing the data to be processed according to a preset mode.
5. The method of claim 4, wherein the data stored by the node server has corresponding data time information, and the identifying the data to be processed in the data stored by the node server according to the critical value includes:
determining a time point corresponding to the critical value;
and identifying the data of which the data time information is smaller than the time point corresponding to the critical value as the data to be processed in the data stored in the node server.
6. The method of claim 1 or 2 or 3 or 5, further comprising:
acquiring target data obtained through processing according to the critical value, and reporting the target data to the central server;
receiving critical data returned by the central server aiming at the target data, and determining a target packet to which the critical data belongs;
And processing the data in the target packet according to a preset mode.
7. A data processing method, applied to a central server, the method comprising:
transmitting a data summarization instruction to a plurality of node servers associated with the central server;
receiving summarized data returned by the plurality of node servers according to the data summarizing instruction, wherein the summarized data is obtained by grouping and summarizing stored data by each node server, and the summarized data comprises data of each group and data of each group;
determining a critical value according to summarized data returned by the plurality of node servers, wherein the critical value is determined according to a sequencing result of the summarized data and a preset data limit, and each node server selects data of data to be processed and data equal to the data limit according to the critical value;
and sending the critical value to the plurality of node servers to instruct the plurality of node servers to process the stored data according to the critical value.
8. The method of claim 7, wherein the data summarization instructions include corresponding grouping information including grouping time periods, wherein determining the threshold from summarized data returned by the plurality of node servers comprises:
Ordering the summarized data returned by the plurality of node servers according to the grouping time period;
adding the sorted summarized data one by one;
and when the added numerical values meet preset limiting conditions, identifying the time period corresponding to the numerical values as a critical value of the data to be processed.
9. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the server is a node server or a central server; if the server is a node server, the processor, when executing the computer program, implements the method of any of claims 1-6; if the server is a central processor, the processor implements the method according to any of claims 7-8 when executing the computer program.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010054694.2A CN111259045B (en) | 2020-01-17 | 2020-01-17 | Data processing method, device, server and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010054694.2A CN111259045B (en) | 2020-01-17 | 2020-01-17 | Data processing method, device, server and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111259045A CN111259045A (en) | 2020-06-09 |
| CN111259045B true CN111259045B (en) | 2023-11-10 |
Family
ID=70950810
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010054694.2A Active CN111259045B (en) | 2020-01-17 | 2020-01-17 | Data processing method, device, server and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111259045B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109033283A (en) * | 2018-07-12 | 2018-12-18 | 广州市闲愉凡生信息科技有限公司 | A kind of distributed search methods of cloud computing platform |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
-
2020
- 2020-01-17 CN CN202010054694.2A patent/CN111259045B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109033283A (en) * | 2018-07-12 | 2018-12-18 | 广州市闲愉凡生信息科技有限公司 | A kind of distributed search methods of cloud computing platform |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111259045A (en) | 2020-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109104336B (en) | Service request processing method and device, computer equipment and storage medium | |
| US10558498B2 (en) | Method for scheduling data flow task and apparatus | |
| CN103748579B (en) | Data are handled in MapReduce frame | |
| CN108111554B (en) | Control method and device for access queue | |
| CN108112038B (en) | Method and device for controlling access flow | |
| CN105989163A (en) | Data real-time processing method and system | |
| CN106713396A (en) | Server scheduling method and system | |
| CN113419856B (en) | Intelligent current limiting method, device, electronic equipment and storage medium | |
| CN111651595A (en) | Abnormal log processing method and device | |
| WO2018166145A1 (en) | Method and device for batch offering of repayment data | |
| CN111796937B (en) | Memory-based resource allocation method, computer equipment and storage medium | |
| CN112837059A (en) | Payment strategy calling method for block chain security protection and digital financial platform | |
| CN113034171A (en) | Business data processing method and device, computer and readable storage medium | |
| CN112769943A (en) | Service processing method and device | |
| CN110321364B (en) | Transaction data query method, device and terminal of credit card management system | |
| CN110490734B (en) | Transaction group construction and broadcasting method and system, equipment and storage medium | |
| CN109189578A (en) | Storage server distribution method, device, management server and storage system | |
| CN109544347B (en) | Tail difference distribution method, computer readable storage medium and tail difference distribution system | |
| CN111259045B (en) | Data processing method, device, server and medium | |
| US11243979B1 (en) | Asynchronous propagation of database events | |
| CN116319810A (en) | Flow control method, device, equipment, medium and product of distributed system | |
| CN114020420A (en) | Distributed to-be-executed task execution method and system, storage medium and terminal | |
| CN113783912A (en) | Request distribution method, device and storage medium | |
| CN112447279A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN113326064A (en) | Method for dividing business logic module, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |