CN114676144A - Data aggregation method and device and computing equipment - Google Patents

Data aggregation method and device and computing equipment Download PDF

Info

Publication number
CN114676144A
CN114676144A CN202210233007.2A CN202210233007A CN114676144A CN 114676144 A CN114676144 A CN 114676144A CN 202210233007 A CN202210233007 A CN 202210233007A CN 114676144 A CN114676144 A CN 114676144A
Authority
CN
China
Prior art keywords
data
service
aggregation
service data
real
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210233007.2A
Other languages
Chinese (zh)
Inventor
方思维
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Rajax Network Technology Co Ltd
Original Assignee
Rajax Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Rajax Network Technology Co Ltd filed Critical Rajax Network Technology Co Ltd
Priority to CN202210233007.2A priority Critical patent/CN114676144A/en
Publication of CN114676144A publication Critical patent/CN114676144A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/235—Update request formulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/248—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems

Abstract

The embodiment of the invention discloses a data aggregation method, a data aggregation device and computing equipment, wherein the method comprises the following steps: responding to the service data updating message, inquiring and acquiring real-time service data from a target service database, and acquiring original service data from a primary data table; the first-level data table is used for storing full service data; performing increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to the target service; updating a secondary data table according to the incremental data, and updating the primary data table according to the real-time service data; and the secondary data table is used for storing the incremental aggregation data corresponding to the service data. The embodiment of the invention realizes that the increment aggregation calculation can be carried out on the real-time service data when the service data updating message is received, thereby improving the efficiency of data aggregation.

Description

Data aggregation method and device and computing equipment
Technical Field
The embodiment of the invention relates to the field of data processing, in particular to a data aggregation method, a data aggregation device, computing equipment and a computer storage medium.
Background
At present, with the rapid development and the popularization and application of computers and information technologies, the scale of an industrial application system is rapidly enlarged, data generated by industrial application is explosively increased, and cloud centralized storage, aggregation processing and data analysis are generally adopted at present.
However, in carrying out embodiments of the present invention, the inventors have discovered that: in the prior art, for data aggregation processing, a certain period is generally set to perform full aggregation on stored data to obtain aggregated data, and aggregation processing needs to be performed depending on a time sequence of the data in an aggregation process, so that the full aggregation is easily affected by message disorder, and the accuracy and efficiency of data aggregation are reduced.
Disclosure of Invention
In view of the above problems, embodiments of the present invention are proposed to provide a data aggregation method, apparatus and computing device, computer storage medium that overcome the above problems, or at least partially solve the above problems.
According to an aspect of an embodiment of the present invention, there is provided a data aggregation method, including:
responding to the service data updating message, inquiring and acquiring real-time service data from a target service database, and acquiring original service data from a primary data table; the primary data table is used for storing full service data;
performing increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to the target service;
updating a secondary data table according to the incremental data, and updating the primary data table according to the real-time service data; and the secondary data table is used for storing the incremental aggregation data corresponding to the service data.
In an optional manner, the service data update message includes target service information; the responding to the service data updating message, inquiring and acquiring real-time service data from the target service database, and acquiring original service data from the primary data table, including: inquiring and acquiring the real-time service data in a target service database according to the target service information; the target service database is a database corresponding to the service side; and acquiring the original service data in the primary data table according to the target service information.
In an optional manner, the performing incremental aggregation calculation according to the real-time service data and the original service data to obtain incremental data corresponding to a target service includes: generating an aggregation message according to the real-time service data and the original service data; and performing increment aggregation calculation according to the aggregation message to obtain increment data corresponding to the target service.
In an optional manner, the aggregation message includes first critical data in original service data and second critical data in the real-time service data; performing incremental aggregation calculation according to the real-time service data and the original service data to obtain incremental data corresponding to a target service, including: and performing incremental aggregation calculation according to the first key data and the second key data in the aggregation message to obtain incremental data corresponding to the target service.
In an alternative manner, the message body of the aggregated message includes an original data model and a current data model; generating an aggregation message according to the real-time service data and the original service data, including: storing the first critical data in the original data model and the second critical data in the current data model to form the aggregated message.
In an alternative manner, the storing the first critical data in the original data model and the second critical data in the current data model to form the aggregated message includes: determining a key field corresponding to a target service; determining first key data from the original service data according to the key field; and determining second key data from the real-time service data according to the key field.
In an optional manner, the first key data is snapshot data of the original service data; the second key data is snapshot data of the real-time service data.
In an optional manner, after the updating the secondary data table according to the incremental data, the method further includes: performing full-volume data aggregation processing on the full-volume service data in the primary data table according to a preset period to obtain a full-volume data aggregation result; and checking the aggregated data in the secondary data table according to the full data aggregation result.
In an optional manner, after the updating the secondary data table according to the incremental data, the method further includes: generating a data report according to the incremental aggregation data in the secondary data table; and displaying the data report to a user.
In an optional manner, the number of the service data update messages is multiple, and the multiple service data update messages are service data update messages corresponding to multiple target objects; the real-time service data is inquired and obtained from a target service database in response to the service data updating message, and the original service data is obtained from a primary data table; before the primary data table is used for storing the full amount of service data, the method further comprises: receiving the service data updating message; when a plurality of service data updating messages corresponding to the same target object are received at the same time, locking the plurality of service data updating messages corresponding to the target object; consuming one of the business data update messages at a time for one of the target objects.
In an optional manner, when a plurality of service data update messages corresponding to the same target object are received at the same time, consuming one service data update message for one target object at a time includes: when a plurality of service data updating messages corresponding to the same target object are received at the same time, the plurality of service data updating messages corresponding to the same target object are locked through a distributed lock, and all the service data updating messages of the target object are consumed in sequence.
According to another aspect of the embodiments of the present invention, there is provided a data aggregation apparatus, including:
the acquisition module is used for responding to the service data updating message, inquiring and acquiring real-time service data from the target service database and acquiring original service data from the primary data table; the first-level data table is used for storing full service data;
the increment aggregation module is used for performing increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to the target service;
the updating module is used for updating a secondary data table according to the incremental data and updating the primary data table according to the real-time service data; and the secondary data table is used for storing the incremental aggregation data corresponding to the service data.
In an optional manner, the service data update message includes target service information; the obtaining module is further configured to: inquiring and acquiring the real-time service data in a target service database according to the target service information; the target service database is a database corresponding to the service side; and acquiring the original service data in the primary data table according to the target service information.
In an optional manner, the incremental aggregation module is further configured to: generating an aggregation message according to the real-time service data and the original service data; and performing increment aggregation calculation according to the aggregation message to obtain increment data corresponding to the target service.
In an optional manner, the aggregation message includes first critical data in original service data and second critical data in the real-time service data; the incremental aggregation module is further configured to: and performing incremental aggregation calculation according to the first key data and the second key data in the aggregation message to obtain incremental data corresponding to the target service.
In an optional manner, a message body of the message body aggregation message includes an original data model and a current data model; the incremental aggregation module is further configured to: storing the first critical data in the original data model and the second critical data in the current data model to form the aggregated message.
In an optional manner, the incremental aggregation module is further configured to: determining a key field corresponding to the target service; determining first key data from the original service data according to the key field; and determining second key data from the real-time service data according to the key field.
In an optional manner, the first key data is snapshot data of the original service data; the second key data is snapshot data of the real-time service data.
In an optional mode, the full aggregation module is configured to perform full data aggregation processing on full service data in the primary data table according to a preset period to obtain a full data aggregation result; and checking the aggregated data in the secondary data table according to the full data aggregation result.
In an optional manner, the apparatus further comprises: the report generation module is used for generating a data report according to the incremental aggregation data in the secondary data table; and the report display module is used for displaying the data report to a user.
In an optional manner, the number of the service data update messages is multiple, and the multiple service data update messages are service data update messages corresponding to multiple target objects; the device further comprises: a message receiving module, configured to receive the service data update message; when a plurality of service data updating messages corresponding to the same target object are received at the same time, locking the plurality of service data updating messages corresponding to the target object; consuming one of the business data update messages at a time for one of the target objects.
In an optional manner, the message receiving module is further configured to: when a plurality of service data updating messages corresponding to the same target object are received at the same time, the plurality of service data updating messages corresponding to the same target object are locked through a distributed lock, and all the service data updating messages of the target object are consumed in sequence.
According to still another aspect of an embodiment of the present invention, there is provided a computing device including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are communicated with each other through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data aggregation method.
According to another aspect of the embodiments of the present invention, a computer storage medium is provided, where at least one executable instruction is stored, and the executable instruction causes a processor to perform an operation corresponding to the data aggregation method.
According to the data aggregation method and device provided by the embodiment of the invention, when a service data updating message is received, real-time service data corresponding to a target service is acquired in real time, original service data is acquired from a primary data table for storing full service data, corresponding aggregation information corresponding to the target service is generated in real time according to the real-time service data and the original service data, incremental aggregation calculation is further performed according to first key data and second key data in the aggregation information to obtain incremental data corresponding to the target service, finally, the incremental data updating is performed on a secondary data table according to the incremental data, the service data after the incremental updating is stored in the secondary data table in real time, the real-time service data is stored in the primary data table, and the incremental aggregation calculation can be performed on the real-time service data when the service data updating message is received, the efficiency of data aggregation is improved. And because the aggregation message corresponding to the target service is generated in real time and the incremental calculation is carried out according to the aggregation message, the method does not need to depend on the time sequence of the message, and improves the accuracy of data aggregation and the stability of the system.
The foregoing description is only an overview of the technical solutions of the embodiments of the present invention, and the embodiments of the present invention can be implemented according to the content of the description in order to make the technical means of the embodiments of the present invention more clearly understood, and the detailed description of the embodiments of the present invention is provided below in order to make the foregoing and other objects, features, and advantages of the embodiments of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the embodiments of the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 shows a flow diagram of a data aggregation method according to one embodiment of the invention;
FIG. 2 shows a flow diagram of a data aggregation method according to another embodiment of the invention;
FIG. 3 is a diagram illustrating an application environment of a data aggregation method according to another embodiment of the present invention;
FIG. 4 is a schematic diagram of a data aggregation apparatus according to an embodiment of the present invention;
FIG. 5 illustrates a schematic structural diagram of a computing device, according to an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
With the rapid expansion of the scale of the industrial application system, data generated by industrial application is explosively increased, and generally, each data source is transmitted to the cloud data center, and the cloud data center performs data conversion, storage, backup and aggregation statistics on the data of each downstream data source for subsequent query and display. However, as the inventor of the present application finds that, in the aggregation process of data in the prior art, since data update in a data source is performed through message transmission, a data center in the cloud determines to trigger data acquisition according to a message, and performs corresponding business data processing and calculation. However, for a message sending mechanism under a distributed architecture, the time sequence cannot be guaranteed, so that due to the influence of the disorder of the message, the data center at the cloud end cannot perform aggregation calculation in real time, but performs full storage after acquiring data according to the message, and sets a certain period to perform aggregation calculation on the data stored in full before, which causes that the data cannot be subjected to aggregation calculation in real time at the data center, and thus the efficiency is low; and because the disorder nature influence of the message, need to process the disorder nature problem of the message according to the business code when carrying on the aggregate computation, has increased the time that the processing procedure needs, and because it is the whole data processing, cause the requirement to the data computing power higher, cause stability and accuracy of the data system to reduce easily.
Based on this, embodiments of the present invention provide a data aggregation method and apparatus, where when a service data update message is received, real-time service data corresponding to a target service is obtained in real time, original service data is obtained from a primary data table for storing full service data, a corresponding aggregation message corresponding to the target service is generated in real time according to the real-time service data and the original service data, incremental aggregation calculation is further performed according to the aggregation message to obtain incremental data corresponding to the target service, and finally, incremental data update is performed on a secondary data table according to the incremental data, the service data after the incremental update is stored in the secondary data table in real time, and the real-time service data is stored in the primary data table, so that the incremental aggregation calculation can be performed on the real-time service data when the service data update message is received. And because the aggregation message corresponding to the target service is generated in real time, and the incremental calculation is performed according to the first key data and the second key data in the aggregation message, the dependency on the time sequence of the message is not required, and the accuracy of data aggregation and the stability of the system are improved.
Fig. 1 is a flowchart illustrating a data aggregation method according to an embodiment of the present invention, where the data aggregation method is applied to a computing device, and may include various types of devices, such as a server or a server cluster, a distributed computing device, a desktop computer, a notebook computer, a tablet computer, a smart phone, a personal computer, and the like, for example, a data center that may be a distributed cluster device, and the embodiment of the present invention is not limited in particular. As shown in fig. 1, the method comprises the steps of:
Step 101, responding to the service data updating message, inquiring and acquiring real-time service data from the target service database, and acquiring original service data from the primary data table.
In the embodiment of the present invention, the service data update message refers to a message for notifying the service data at the service side of updating, and the service data change message carries target service information. For example, for order service, a new order is generated at a merchant service platform or a message that order data changes is generated, and target service information carried in the message includes information such as an order ID and time.
After receiving the service data update message, responding to the service data update message, and inquiring and acquiring the latest real-time service data from a target service database of a corresponding service side in real time according to target service information, wherein the target service database is a database corresponding to the service side. And simultaneously, acquiring the original service data in a primary data table according to the target service information. The primary data table is used to store the received full service data, and may be a wide table used by the data center itself to store the full service data of each service side, for example.
For a data center of a distributed architecture, the data center can simultaneously receive service data update messages sent by each service side, that is, a plurality of service data update messages are provided, and the plurality of service data update messages are service data update messages corresponding to a plurality of target objects. The same target object at the same service side may also generate multiple service data update messages at the same time, so that the received multiple service data update messages need to be preprocessed, and only one service data update message is consumed for the multiple service data update messages of the same target object at a time. After receiving the service data update message, when receiving a plurality of service data update messages corresponding to the same target object at the same time, the embodiment of the invention locks the plurality of service data update messages corresponding to the target object, and only one service data update message is consumed for one target object each time. Specifically, for a data center with a distributed architecture, when a plurality of service data update messages corresponding to the same target object are received at the same time, the plurality of service data update messages corresponding to the same target object can be locked through a distributed lock, and each service data update message of the target object is consumed in sequence. The target object in the embodiment of the invention is an object for generating real-time service data at a service side, such as an online catering ordering scene, the target object can be a catering store, and the real-time service data is catering order data. According to the embodiment of the invention, the latest real-time service data of the target service is acquired according to the service data updating message, and the latest real-time service data is inquired and acquired in the target service database after the service data updating message is acquired each time, so that the latest real-time service data is acquired regardless of the time sequence order of the service data updating message, and the mode does not need to process data according to the time sequence of the service data updating message, thereby improving the processing efficiency.
In the embodiment of the invention, after the real-time service data corresponding to the target service is acquired and the original service data is acquired from the primary data table, the real-time acquired real-time service data can be stored in the primary data table so as to update the full storage data.
Step 102: and performing increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to the target service.
According to the embodiment of the invention, the aggregation message is generated according to the real-time service data and the original service data, and the increment aggregation calculation is carried out according to the aggregation message to obtain the increment data corresponding to the target service.
The aggregation message comprises first key data in original service data and second key data in the real-time service data. In an embodiment of the present invention, the aggregated message includes a message body, and the message body includes an original data model (orginModel) and a current data model (rtd), where the original data model (orginModel) is used to store first key data of the service data stored in the primary data table before the real-time service data is acquired, that is, the original service data. The current data model (currentModel) is used for storing second key data of real-time service data obtained in real time, storing the first key data in the original data model, and storing the second key data in the current data model to form the aggregated message. Specifically, a key field corresponding to a target service (such as an order service) is determined, first key data is determined from the original service data according to the key field, the first key data is stored in the original data model, second key data is determined from the real-time service data according to the key field, and the second key data is stored in the current data model, so that a message body of an aggregated message is finally obtained. The key field is key data required for performing incremental aggregation on the target service, and the key field is a preset field, and a person skilled in the art can perform corresponding setting according to a specific scenario, which is not limited in the embodiment of the present invention. In an embodiment of the present invention, the first critical data and the second critical data may be snapshot data; specifically, the first key data is snapshot data of the original service data; the second key data is snapshot data of the real-time service data. By using snapshot data, the write-in amount of data can be reduced, and the efficiency of incremental calculation is improved.
After the aggregation message is obtained, performing incremental aggregation calculation according to the first key data and the second key data in the aggregation message to obtain incremental data corresponding to the target service, that is, incremental data Δ ═ currentModel-orginModel.
Step 103: and updating a secondary data table according to the incremental data, and updating the primary data table according to the real-time service data.
And the secondary data table is used for storing the incremental aggregation data corresponding to the service data. After the incremental data are obtained, the incremental aggregation data in the secondary data table can be updated according to the incremental data, and the updated incremental aggregation data are obtained. And meanwhile, updating the primary data table according to the real-time service data. In the embodiment of the invention, when the service data updating message is out of order, the original service data in the primary data table is updated in the previous consumption process of the service data updating message, and the primary data table does not update and store the real-time service data in the consumption process corresponding to the current service data updating message.
In the embodiment of the invention, a catering order scene is taken as an example, when a target service is taken as a dimension of a sum to be settled, for a normal order W, the sum to be settled at the time 1 is 50 yuan, a customer adds dishes at the time 2, the sum to be settled is 100 yuan, a customer adds dishes at the time 3, and the sum to be settled is 200 yuan, according to a normal sequence, a data center firstly receives a service data updating message corresponding to the sum to be settled at the time 1, then receives a service data updating message corresponding to the sum to be settled at the time 2, and finally receives a service data updating message corresponding to the sum to be settled at the time 3.
After receiving the service data update message corresponding to the amount to be settled at time 1, the data center obtains the real-time service data corresponding to the service data update message from the target database (that is, the amount to be settled at time 1 is 50 yuan), and then the initial primary data table stores the real-time service data W1 as 50 since there is no data corresponding to the order ID, and the secondary data table also stores the incremental aggregation data W2 as 50 since there is no data corresponding to the order ID before. After receiving the service data update message corresponding to the amount to be settled at time 2, acquiring corresponding real-time service data "the amount to be settled is 100 yuan" and original service data "W1 ═ 50" in the primary data table, and then performing data update on the order W by using the secondary data table if the calculated increment Δ 1 is 100-W1 ═ 50, and W2 ═ 50 +/Δ 1 ═ 50+50 ═ 100; in the same manner, after receiving the service data updating message corresponding to the amount to be settled at the time 3, the increment Δ 2 is 200-; after the second data update for order W, the secondary table calculates the increment W2 ═ 100 +. DELTA.2 ═ 100+100 ═ 200.
When the service data update message received by the data center is out of order, for example, the service data update message corresponding to the amount to be settled at time 1 is received first, then the service data update message corresponding to the amount to be settled at time 3 is received, and finally the service data update message corresponding to the amount to be settled at time 2 is received, after the service data update message corresponding to the amount to be settled at time 1 is received, the data center obtains the latest real-time service data corresponding to the order W from the target database (that is, the amount to be settled at time 1 is 50 yuan), if no data corresponding to the order ID exists in the initial primary data table, the real-time service data W1 is stored as 50, and if no data corresponding to the order ID exists in the secondary data table, the incremental aggregation data W2 is also stored as 50. After receiving the service data update message corresponding to the amount to be settled at time 3, querying the latest real-time service data "the amount to be settled is 200 yuan" in the target database and the original service data "W1 is 50" in the primary data table, if the calculation increment Δ 1 is 200-W1 is 150, then the secondary data table performs data update for the order W, and if the calculation increment W2 is 50+ Δ 1 is 50+150 is 200, and at the same time the primary data table determines that the real-time service data is the latest data, then "the amount to be settled is 200 yuan" is stored in the primary data table. In the same way, after receiving the service data update message corresponding to the amount to be settled at the time 2, querying that the latest real-time service data of the order W in the target database is ' the amount to be settled is 200 yuan ' and the original service data ' W1 in the primary data table is 200-; the secondary data table is updated for the second time of order W and the increment W2-200 + Δ 2-100-200 is calculated. By the mode, when the service data updating message is received, the latest real-time service data is inquired from the target service database in real time, and the incremental calculation is carried out according to the real-time service data and the original service data, so that the full amount of service data in the primary data table can be updated, the real-time incremental aggregation data processing can be synchronously carried out, and the real-time incremental calculation is realized. In the processing process, processing is not needed according to the time sequence of the service data updating message, and processing is only carried out according to the latest real-time service data and the original service data corresponding to the target service, so that the processing does not need to depend on the time sequence of the message, and the accuracy of data aggregation and the stability of the system are improved.
According to the data aggregation method provided by the embodiment of the invention, when a service data updating message is received, real-time service data corresponding to a target service is obtained in real time, original service data is obtained from a primary data table for storing full-volume service data, a corresponding aggregation message corresponding to the target service is generated in real time according to the real-time service data and the original service data, incremental aggregation calculation is further performed according to first key data and second key data in the aggregation message to obtain incremental data corresponding to the target service, finally, the incremental data updating is performed on a secondary data table according to the incremental data, the service data after the incremental updating is stored in the secondary data table in real time, the real-time service data is stored in the primary data table, and the incremental aggregation calculation can be performed on the real-time service data when the service data updating message is received, the efficiency of data aggregation is improved. And because the aggregation message corresponding to the target service is generated in real time, and the incremental calculation is performed according to the first key data and the second key data in the aggregation message, the dependency on the time sequence of the message is not required, and the accuracy of data aggregation and the stability of the system are improved.
Fig. 2 shows a flow diagram of a data aggregation method according to another embodiment of the invention. First, an application environment of the embodiment of the present invention will be described. Referring to fig. 3, fig. 3 is a schematic diagram illustrating an application environment of a data aggregation method according to another embodiment of the present invention. The method is applied to a data center of a distributed cluster. The data center is used for storing order data, and comprises a data supply application, a service inquiry application, a data aggregation application, a primary data table and a secondary data table. The order supply application is used for acquiring service data updating messages sent by all downstream systems and sending the messages to the service inquiry application. The service query application is used for querying real-time service data corresponding to the target service, and the data aggregation application is used for aggregating the service data. Referring to fig. 2 and fig. 3, a data aggregation method according to an embodiment of the present invention includes the following steps:
step 201: the data feed application obtains order data update messages sent by downstream systems.
The downstream system sends an order data updating message of each target service system to a data supply application of the data center through the downstream system of the distributed cluster, wherein the data updating message carries target order information, and the target order information comprises a target order ID, corresponding order data information and the like. The downstream system is a downstream system of a distributed cluster, and may be a PaaS side, for example. Cloud computing Service platforms can be divided into three layers, i aas (Infrastructure) -as-a-Service), PaaS (Platform) -as-a-Service), and SaaS (Software) -as-a-Service. The first layer, IaaS, infrastructure as a Service, also known as Hardware-as-a-Service, obtains services, such as storage and databases, from a sophisticated computer infrastructure over the Internet. The second layer PaaS, i.e., platform as a service, provides a corresponding server platform or development environment as a service, which becomes PaaS (platform as a service). PaaS actually refers to a platform developed by software as a service, which is submitted to a user in a SaaS mode. PaaS is therefore also an application of the SaaS model. The third layer SaaS is a short for software as a service, and is a mode for providing software through the Internet, a manufacturer uniformly deploys application software on a server of the manufacturer, and a customer can order a required application software service to the manufacturer through the Internet according to actual requirements of the manufacturer. Wherein the data-feed application ensures that a subsequent message from the same store is being consumed by adding a distributed lock.
Step 201: and the data supply application updates the message according to the order data, sends the message to the service inquiry application so as to obtain the latest real-time order data and obtain the corresponding original order data from the primary data table.
And the data supply application immediately sends the message to the service inquiry application after receiving the order data updating message. The service inquiry application will inquire the latest real-time service data in the target service database after receiving the order data updating message each time. And the service inquiry application updates the target order ID in the message according to the order data, inquires the latest real-time service data corresponding to the target order ID in a target service database and feeds back the latest real-time service data to the data supply application.
Step 202: and the data supply application generates an aggregation message according to the acquired real-time order data and the original order data, sends the aggregation message to the data aggregation application, and updates the primary data table according to the real-time order data.
In the embodiment of the present invention, the process of generating the aggregation message by the data providing application according to the obtained real-time order data and the original order data is substantially the same as the process of generating the aggregation message according to the obtained real-time service data and the original service data in the embodiment described above, and details are not repeated here.
Wherein updating the primary data table according to the real-time order data comprises: due to the influence of message disorder, when the real-time order data is consistent with the original order data in the primary data table, the original order data in the primary data table is not updated.
Step 203: and the data aggregation application performs increment aggregation calculation according to the aggregation message to obtain increment data corresponding to the target order.
The process of obtaining the incremental data corresponding to the target order by the data aggregation application according to the aggregation message in the embodiment of the present invention is substantially the same as that in the above embodiment, and details are not repeated here.
Step 204: and the data aggregation application sends the incremental data to the secondary data table so that the secondary data table is updated according to the incremental data.
The data aggregation application updates and accumulates the current data of the secondary table by overlapping increment delta each time, and the data is updated according to the increment delta, so that the data aggregation does not need to pay attention to the time sequence.
According to the data aggregation method provided by the embodiment of the invention, when a service data updating message is received, real-time service data corresponding to a target service is acquired in real time, original service data is acquired from a primary data table for storing full service data, a corresponding aggregation message corresponding to the target service is generated in real time according to the real-time service data and the original service data, incremental aggregation calculation is further performed according to first key data and second key data in the aggregation message to obtain incremental data corresponding to the target service, finally, the incremental data updating is performed on a secondary data table according to the incremental data, the service data after the incremental updating is stored in the secondary data table in real time, the real-time service data is stored in the primary data table, and the incremental aggregation calculation of the real-time service data can be performed when the service data updating message is received, the efficiency of data aggregation is improved. And because the aggregation message corresponding to the target service is generated in real time and incremental calculation is performed according to the first key data and the second key data in the aggregation message, the method does not need to depend on the time sequence of the message, and the accuracy of data aggregation and the stability of the system are improved.
Fig. 4 shows a schematic structural diagram of a data aggregation apparatus according to an embodiment of the present invention. As shown in fig. 4, the apparatus 400 includes: an acquisition module 410, an incremental aggregation module 420, and an update module 430.
An obtaining module 410, configured to query and obtain real-time service data from a target service database in response to a service data update message, and obtain original service data from a primary data table; the primary data table is used for storing full service data;
the increment aggregation module 420 is configured to perform increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to a target service;
the updating module 430 is configured to update the secondary data table according to the incremental data, and update the primary data table according to the real-time service data; and the secondary data table is used for storing the incremental aggregation data corresponding to the service data.
In an optional manner, the service data update message includes target service information; the obtaining module 410 is further configured to: inquiring and acquiring the real-time service data in a target service database according to the target service information; the target service database is a database corresponding to the service side; and acquiring the original service data in the primary data table according to the target service information.
In an optional manner, the incremental aggregation module 420 is further configured to: generating an aggregation message according to the real-time service data and the original service data; and performing increment aggregation calculation according to the aggregation message to obtain increment data corresponding to the target service.
In an optional manner, the aggregation message includes first critical data in original service data and second critical data in the real-time service data; the incremental aggregation module 420 is further configured to: and performing incremental aggregation calculation according to the first key data and the second key data in the aggregation message to obtain incremental data corresponding to the target service.
In an optional manner, a message body of the message body aggregation message includes an original data model and a current data model; the incremental aggregation module 420 is further configured to: storing the first critical data in the original data model and the second critical data in the current data model to form the aggregated message.
In an optional manner, the incremental aggregation module 420 is further configured to: determining a key field corresponding to the target service; determining first key data from the original service data according to the key field; and determining second key data from the real-time service data according to the key field.
In an optional manner, the first key data is snapshot data of the original service data; the second key data is snapshot data of the real-time service data.
In an optional mode, the full aggregation module is configured to perform full data aggregation processing on full service data in the primary data table according to a preset period to obtain a full data aggregation result; and checking the aggregated data in the secondary data table according to the full data aggregation result.
In an optional manner, the apparatus further comprises: the report generation module is used for generating a data report according to the incremental aggregation data in the secondary data table; and the report display module is used for displaying the data report to a user.
In an optional manner, the number of the service data update messages is multiple, and the multiple service data update messages are service data update messages corresponding to multiple target objects; the device further comprises: a message receiving module, configured to receive the service data update message; when a plurality of service data updating messages corresponding to the same target object are received at the same time, locking the plurality of service data updating messages corresponding to the target object; consuming one of the business data update messages at a time for one of the target objects.
In an optional manner, the message receiving module is further configured to: when a plurality of service data updating messages corresponding to the same target object are received at the same time, the plurality of service data updating messages corresponding to the same target object are locked through a distributed lock, and all the service data updating messages of the target object are consumed in sequence.
The descriptions of the above modules of the data aggregation apparatus 400 provided in the embodiment of the present invention refer to the corresponding descriptions in the method embodiment, and are not repeated herein.
According to the data aggregation device provided by the embodiment of the invention, when a service data updating message is received, real-time service data corresponding to a target service is acquired in real time, original service data is acquired from a primary data table for storing full service data, a corresponding aggregation message corresponding to the target service is generated in real time according to the real-time service data and the original service data, incremental aggregation calculation is further performed according to first key data and second key data in the aggregation message to obtain incremental data corresponding to the target service, finally, the incremental data updating is performed on a secondary data table according to the incremental data, the service data after the incremental updating is stored in the secondary data table in real time, the real-time service data is stored in the primary data table, and the incremental aggregation calculation of the real-time service data can be performed when the service data updating message is received, the efficiency of data aggregation is improved. And because the aggregation message corresponding to the target service is generated in real time and incremental calculation is performed according to the first key data and the second key data in the aggregation message, the method does not need to depend on the time sequence of the message, and the accuracy of data aggregation is improved.
The embodiment of the present invention further provides a non-volatile computer storage medium, where the computer storage medium stores at least one executable instruction, and the executable instruction may execute the data aggregation method in any of the method embodiments.
Fig. 5 is a schematic structural diagram of a computing device according to an embodiment of the present invention, where the specific embodiment of the present invention does not limit a specific implementation of the computing device.
As shown in fig. 5, the computing device may include: a processor (processor)502, a Communications Interface (Communications Interface)504, a memory 506, and a communication bus 508.
Wherein:
the processor 502, communication interface 504, and memory 506 communicate with one another via a communication bus 508.
A communication interface 504 for communicating with network elements of other devices, such as clients or other servers.
The processor 502 is configured to execute the program 510, and may specifically execute relevant steps in the above data aggregation method embodiments.
In particular, program 510 may include program code that includes computer operating instructions.
The processor 502 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
A memory 506 for storing a program 510. The memory 506 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 510 may specifically be configured to cause the processor 502 to execute the data aggregation method in any of the above-described method embodiments. For specific implementation of each step in the program 510, reference may be made to corresponding steps and corresponding descriptions in units in the foregoing data aggregation method embodiments, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. In addition, embodiments of the present invention are not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of embodiments of the present invention as described herein, and any descriptions of specific languages are provided above to disclose preferred embodiments of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that is, the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the devices in an embodiment may be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore, may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Moreover, those of skill in the art will appreciate that while some embodiments herein include some features included in other embodiments, not others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components according to embodiments of the present invention. Embodiments of the invention may also be implemented as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing embodiments of the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website, or provided on a carrier signal, or provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of the invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names. The steps in the above embodiments should not be construed as limiting the order of execution unless specified otherwise.

Claims (10)

1. A method of data aggregation, comprising:
responding to the service data updating message, inquiring and acquiring real-time service data from a target service database, and acquiring original service data from a primary data table; the primary data table is used for storing full service data;
performing increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to the target service;
updating a secondary data table according to the incremental data, and updating the primary data table according to the real-time service data; and the secondary data table is used for storing the incremental aggregation data corresponding to the service data.
2. The method of claim 1, wherein,
the service data updating message comprises target service information;
the responding to the service data updating message, inquiring and acquiring real-time service data from the target service database, and acquiring original service data from the primary data table, comprises:
inquiring and acquiring the real-time service data in a target service database according to the target service information;
and acquiring the original service data in the primary data table according to the target service information.
3. The method of claim 1, wherein,
the performing incremental aggregation calculation according to the real-time service data and the original service data to obtain incremental data corresponding to a target service includes:
generating an aggregation message according to the real-time service data and the original service data;
and performing increment aggregation calculation according to the aggregation message to obtain increment data corresponding to the target service.
4. The method of claim 3, wherein the aggregated message comprises first critical data in original traffic data and second critical data in the real-time traffic data;
the performing incremental aggregation calculation according to the real-time service data and the original service data to obtain incremental data corresponding to a target service includes:
and performing incremental aggregation calculation according to the first key data and the second key data in the aggregation message to obtain incremental data corresponding to the target service.
5. The method of claim 4, wherein a message body of the aggregated message includes an original data model and a current data model;
generating an aggregation message according to the real-time service data and the original service data, including:
Storing the first critical data in the original data model and the second critical data in the current data model to form the aggregated message.
6. The method of claim 5, wherein,
the storing the first critical data in the original data model and the second critical data in the current data model to form the aggregated message comprises:
determining a key field corresponding to a target service;
determining first key data from the original service data according to the key field;
and determining second key data from the real-time service data according to the key field.
7. The method according to any of claims 4-6, wherein the first critical data is snapshot data of the original traffic data; the second key data is snapshot data of the real-time service data.
8. A data aggregation apparatus, comprising:
the acquisition module is used for responding to the service data updating message, inquiring and acquiring real-time service data from the target service database and acquiring original service data from the primary data table; the primary data table is used for storing full service data;
The increment aggregation module is used for performing increment aggregation calculation according to the real-time service data and the original service data to obtain increment data corresponding to the target service;
the updating module is used for updating a secondary data table according to the incremental data and updating the primary data table according to the real-time service data; and the secondary data table is used for storing the incremental aggregation data corresponding to the service data.
9. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface are communicated with each other through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the data aggregation method in any one of claims 1-7.
10. A computer storage medium having stored therein at least one executable instruction that causes a processor to perform operations corresponding to the data aggregation method of any one of claims 1-7.
CN202210233007.2A 2022-03-09 2022-03-09 Data aggregation method and device and computing equipment Pending CN114676144A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210233007.2A CN114676144A (en) 2022-03-09 2022-03-09 Data aggregation method and device and computing equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210233007.2A CN114676144A (en) 2022-03-09 2022-03-09 Data aggregation method and device and computing equipment

Publications (1)

Publication Number Publication Date
CN114676144A true CN114676144A (en) 2022-06-28

Family

ID=82072331

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210233007.2A Pending CN114676144A (en) 2022-03-09 2022-03-09 Data aggregation method and device and computing equipment

Country Status (1)

Country Link
CN (1) CN114676144A (en)

Similar Documents

Publication Publication Date Title
CN108153670B (en) Interface testing method and device and electronic equipment
US20220343398A1 (en) Order management methods, system, terminal and electronic device based on multi-person ordering
CN111639965B (en) Integral updating method and device
US20060020520A1 (en) Systems and methods for processing electronic documents in a computer network
WO2020033553A1 (en) System and method for anomaly detection and deduplication of electronic data feeds
CN109254854A (en) Asynchronous invoking method, computer installation and storage medium
CN111125496A (en) Price query method, device and system
CN107133788A (en) A kind of reimbursement processing method and processing device
CN112884405A (en) Inquiry system and scheduling method thereof
CN111813868B (en) Data synchronization method and device
CN112288545A (en) Information processing method, information transmitting method, information updating method, information processing apparatus, information updating apparatus, and information updating medium
CN111159211A (en) Order information generation method, device and system, computer equipment and storage medium
CN111445340A (en) Method and device for estimating value of financial product, electronic equipment and readable storage medium
US20150310390A1 (en) Aggregation and workflow engines for managing project information
US11297156B1 (en) System for processing data using distributed messaging system and data processing method thereof
CN113592462A (en) Online delivery method, system, electronic device and medium
CN111626806B (en) Data processing method and device and electronic equipment
CN111897826A (en) Parameter information updating method and device, electronic equipment and readable storage medium
CN114676144A (en) Data aggregation method and device and computing equipment
CN110765131A (en) Data compression method and device for goods source data, computer equipment and storage medium
JP2020057356A (en) Intelligent prediction of bundles of spare parts
US20240054451A1 (en) Inventory information processing method and device
CN112465599B (en) Order processing method, order processing system and computing equipment
CN114820132A (en) Order distribution method and device, electronic equipment and storage medium
CN113763100B (en) Order processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination