CN111459901A - Data aggregation calculation method, device and equipment - Google Patents
Data aggregation calculation method, device and equipment Download PDFInfo
- Publication number
- CN111459901A CN111459901A CN202010302403.7A CN202010302403A CN111459901A CN 111459901 A CN111459901 A CN 111459901A CN 202010302403 A CN202010302403 A CN 202010302403A CN 111459901 A CN111459901 A CN 111459901A
- Authority
- CN
- China
- Prior art keywords
- data
- calculation
- aggregation
- aggregation calculation
- aggregated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 212
- 230000002776 aggregation Effects 0.000 title claims abstract description 149
- 238000004220 aggregation Methods 0.000 title claims abstract description 149
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000007906 compression Methods 0.000 claims description 44
- 230000006835 compression Effects 0.000 claims description 44
- 230000009467 reduction Effects 0.000 claims description 42
- 238000001914 filtration Methods 0.000 claims description 8
- 230000004931 aggregating effect Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000008520 organization Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 10
- 239000000047 product Substances 0.000 description 10
- 230000007246 mechanism Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000011156 evaluation Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000006227 byproduct Substances 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
Abstract
A method, a device and equipment for calculating data aggregation are disclosed. Determining an aggregation calculation protocol for the data to be aggregated which needs to be calculated, dynamically obtaining calculation attributes which need to be used during calculation, reducing the dimension of the data to be aggregated, further compressing the dimension-reduced data, and then performing aggregation calculation by using the compressed dimension-reduced data.
Description
Technical Field
The embodiment of the specification relates to the technical field of information, in particular to a method, a device and equipment for calculating aggregation of data.
Background
In many fields, it is often necessary to perform aggregate calculations on data to obtain corresponding results. However, in practical applications, the data to be aggregated of one target object tends to be huge in quantity. For example, in asset valuation for an organization, there may be millions of asset records under the name of an organization or a conglomerate, requiring considerable computational effort if aggregation is required.
Based on this, there is a need for a more efficient aggregation calculation scheme.
Disclosure of Invention
The embodiment of the application aims to provide an aggregation calculation scheme with higher efficiency.
In order to solve the above technical problem, the embodiment of the present application is implemented as follows:
a method of aggregate computation of data, comprising:
acquiring a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes;
determining an aggregation calculation protocol adopted by a target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula;
determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
determining a compression condition, wherein the compression condition comprises a part of calculation attributes, compressing the rest part of calculation attributes in the dimension reduction data according to the compression condition, and generating compressed dimension reduction data;
and performing aggregation calculation according to the aggregation calculation formula and the compressed dimension reduction data to generate an aggregation calculation result aiming at the target object.
Correspondingly, the embodiment of the present specification further provides a data aggregation computing device, including:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module acquires a plurality of pieces of data to be aggregated of a target object, and the data to be aggregated comprises a plurality of fixed attributes;
the determining module is used for determining an aggregation calculation protocol adopted by a target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula;
the filtering module is used for determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
the compression module is used for determining a compression condition, wherein the compression condition comprises a part of calculation attributes, compressing the rest calculation attributes in the dimensionality reduction data according to the compression condition and generating compressed dimensionality reduction data;
and the aggregation calculation module is used for performing aggregation calculation according to the aggregation calculation formula and the compressed dimension reduction data to generate an aggregation calculation result aiming at the target object.
According to the scheme provided by the embodiment of the specification, based on the pre-established multidimensional model, an aggregation calculation protocol is determined for the data to be aggregated which needs to be calculated, the calculation attribute which needs to be used in calculation is dynamically obtained, the dimension of the data to be aggregated is reduced, the dimension-reduced data is further compressed, then the compressed dimension-reduced data is used for aggregation calculation, invalid judgment and calculation are reduced, the calculation amount in aggregation is reduced, and efficient aggregation calculation of large-scale data is realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of embodiments of the invention.
In addition, any one of the embodiments in the present specification is not required to achieve all of the effects described above.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present specification, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1 is a schematic flow chart of a method for calculating aggregation of data according to an embodiment of the present disclosure;
FIG. 2 is a diagram illustrating dimension reduction data according to an embodiment of the present disclosure;
FIG. 3 is a diagram illustrating compressed dimension reduced data according to an embodiment of the present disclosure;
FIG. 4 is a schematic structural diagram of a data aggregation computing device according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an apparatus for configuring a method according to an embodiment of the present disclosure.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present specification, the technical solutions in the embodiments of the present specification will be described in detail below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all the embodiments. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of protection.
Performing aggregate computations on large-scale data is a current scenario that is often faced. Such as transaction statistics, asset valuations, and the like. Aggregate computation refers to performing computation on a set of values and returning a single computation, including computing means such as averaging, summing, extremum, counting, etc.
Taking asset valuation as a specific example, securitization of assets is a commonly used means at present, so valuation of assets is also a necessary means in financial activities, which often requires aggregation of asset records of an organization, which in turn contains invalid asset records. In practical applications, many organizations tend to have huge assets, for example, for multi-organization billion-level assets, one of the assets in the organizations may be as large as millions, and in such a situation, if it is desired to achieve efficient real-time aggregate valuation, a considerable number of servers are required to perform parallel computations, which is not practical and is inefficient. Based on this, the embodiment of the present specification provides an efficient data aggregation calculation method.
The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings. As shown in fig. 1, fig. 1 is a schematic flow chart of a data aggregation calculation method provided in an embodiment of this specification, where the flow chart specifically includes the following steps:
s101, acquiring a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes.
The target objects are different depending on the implementation scenario. For example, under the transaction statistics of the e-commerce platform, the target object may be a certain merchant; in the scenario of asset valuation, the target object may be a certain enterprise or a group.
In an application scenario of asset estimation, data to be aggregated may be asset records in a large-scale asset pool including a plurality of organizations, and each piece of data to be aggregated has a corresponding target object. For example, a financing institution which qualifies for loan issues a record of property business data formed by various loans in a securitization supported by the property to which its own project belongs. In an asset pool, there are often a large number of institutions participating, each under the name of which there is a corresponding securitized asset to sell, at which time asset valuation may be required if other buyers want to buy an institution's assets. At this time, such a purchasing-intentioned buyer organization may be regarded as an associated organization of data to be aggregated, and obviously, one or more associated organizations may exist.
Since the asset pool is pre-existing, in other words, before that, for any asset of each organization, a bottom model applied to asset description can be pre-constructed at the server, and the bottom model can include multiple dimensions (i.e. fixed attributes), so that the attributes and features of an asset data record are described, and theoretically, the attributes and features can be combined at will, so that any asset data to be evaluated can be described by using a data record including fixed attributes. As shown in table 1, table 1 is a fixed dimension schematic table for dimension depiction of an asset provided in the embodiments of the present specification.
By depicting any asset through the dimensions, each piece of asset data to be evaluated can be endowed with corresponding fixed attributes, the value under each fixed attribute is determined, and a piece of data to be aggregated containing a plurality of fixed attributes and the fixed attribute values thereof is generated.
It should be noted that, in this manner, a piece of data to be aggregated that is depicted may include a considerable number of dimensions, for example, if a total of 30 dimensions are provided in a preset model, each piece of data may have 30 fixed dimensions, and values of some of the fixed dimensions may be invalid values (e.g., default values, or blank values).
In other scenarios, for example, in the scenario of transaction statistics, the fixed attribute may have other dimensions, for example, in the scenario of transaction statistics, the fixed attribute that a transaction may include may be an attribute such as "product code", "purchase object", "buyer age", "transaction amount", and the like.
In practical applications, it may be that the fixed attributes included in the data to be aggregated are different from each other or only partially identical. For example, asset data to be valued 1 includes asset dimension 1 through asset dimension 10, while asset data to be valued 2 includes asset dimension 5 through asset dimension 20.
Furthermore, in practical applications, for convenience of statistics, classification and storage may be performed according to a target object (or a target object to which the target object belongs) included in one piece of data to be aggregated. For example, in a relational database, a target organization included in asset data to be evaluated is stored as a primary key. Thereby facilitating the overall valuation of all assets of a target organization by other users.
Based on the foregoing, any associated entity (i.e., buyer, which may be an individual user or an entity user) may obtain a plurality of asset data to be valued for a target entity from the pool of assets based on the identity of the target entity. In the process, a plurality of pieces of asset data to be evaluated obtained in a return mode can be further screened based on certain fixed asset attributes.
S103, determining an aggregation calculation protocol adopted by the target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula.
In the bottom model of the server, a plurality of calculation formulas for calculating one piece of data to be aggregated can be also preset, and an aggregation calculation formula adapted to a certain target mechanism or a certain class of target mechanisms is given, so that a plurality of predefined aggregation calculation protocols are formed. For example, in the scenario of asset valuation, it may be called valuation protocol, as shown in table 2, table 2 is a schematic table of an aggregation calculation protocol given in the embodiments of the present specification.
PV, P, I, F, R, and other characters, etc. in the table all characterize some fixed attribute in the data to be aggregated. In an aggregation calculation protocol, at least one aggregation calculation formula for calculation should be included, and of course, a plurality of calculation formulas may be included.
When the fixed attributes and the values thereof contained in the data to be aggregated conform to the aggregation calculation formula in the table, the corresponding aggregation calculation formula can be used to perform aggregation calculation on the data with aggregation.
And S105, determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering the fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula.
The calculation attribute refers to a fixed attribute required to be adopted in an aggregation calculation formula. Taking formula 2 in table 2 as an example, the calculation attributes included therein are "PV", "P", "V", and "I".
In each aggregation calculation formula, it is necessary to include the calculation attribute for the calculation formula (i.e., the fixed attribute for calculating the formula), and it is easy to understand that, for a calculation formula, which generally includes a limited number of calculation attributes, the number of calculation attributes is much smaller than the fixed attribute.
For example, in a model, there may be 50 solid yield attributes for the total quantity, while the number of calculated attributes in a particular aggregate calculation formula may not exceed 5. Thus, for each aggregate calculation formula, there will be a set of attributes that correspond to the calculation attributes that the aggregate calculation formula needs to employ.
When the aggregation calculation protocol has a plurality of aggregation calculation formulas, the calculation attribute to be adopted is the union set of the attribute sets corresponding to the aggregation calculation formulas.
Furthermore, dimension reduction can be performed on the data to be aggregated according to the plurality of calculation attributes obtained by determination, and invalid fixed asset dimensions (i.e. dimensions that cannot be adopted in the aggregation calculation protocol) are filtered out, so that dimension reduction data only containing the calculation attributes is generated.
As described above, because the fixed attributes included in the data to be aggregated may be different from each other or only partially the same, the dimensions of the data to be aggregated can be unified by the foregoing dimension reduction method, so that the data dimensions are unified.
S107, determining a compression condition, wherein the compression condition comprises a part of calculation attributes, compressing the rest calculation attributes in the dimension reduction data according to the compression condition, and generating the compressed dimension reduction data.
As mentioned above, at this time, each data to be aggregated has been filtered out by several dimensions, and unification on data dimensions is achieved, but the actual number is not reduced. As shown in fig. 2, fig. 2 is a schematic diagram of dimension reduction data according to an embodiment of the present disclosure.
For example, in a relational database, the data may be compressed using group by in a Structured Query language (SQ L).
For example, the compression conditions are used for the data in fig. 2: sql "select sum, penalty, sum from the platform table where the target organization = 'XXA' group by product code, five-level classification, repayment mode" performs data compression, thereby obtaining the compressed dimension reduction data as shown in fig. 3. The partial calculation attributes are product code, five-level classification and repayment mode, and the rest partial calculation attributes are principal, interest, penalty and expense.
Based on the foregoing compression method, it is thereby achieved that a plurality of pieces of data having the same value of the calculation attribute are compressed into one data record, although in the schematic diagram, only 4 records are compressed into two pieces of data. However, in practical applications, since the number of asset records having the same calculation attribute value may be quite large in the same target organization, tens of millions of assets of a certain organization can be compressed to hundreds of assets in practical applications, and the compression rate is quite large.
In the scenario of transaction statistics, when it is necessary to count transactions of users of certain products in various age groups in various time periods, corresponding compression needs to be performed based on the "time period" and the "age".
In practical application, which compression conditions need to be selected for compression specifically, in one embodiment, pre-customization may be performed according to selection of a user, and in another embodiment, the remaining calculation attributes in the data to be aggregated may be aggregated according to the calculation asset attributes included in the adaptation conditions and according to the applicable conditions required by the aggregation calculation pricing formula in the aggregation calculation protocol, so that in this manner, the data obtained by aggregation is the compressed data corresponding to the aggregation calculation formula. Namely, the adaptive condition of the aggregation calculation formula is used as the compression condition,
for example, for the calculation formula 1 in table 2, the adaptation condition is "product code =001, five-level classification = normal", the data records of "product code =001, five-level classification = normal" in the calculation asset attribute may be aggregated, so as to obtain the compressed data adapted to the calculation formula 1.
In other words, the "determining the compression condition" herein may need to determine not only the calculation attribute to be adopted during the compression, but also the value of the calculation attribute. For example, in this case, if "product code =001, five-level classification = normal" is not included in the calculation attribute of one data record, then subsequent compression is not involved.
In this way, some invalid data can be filtered out, for example, in an evaluation scenario, for asset data records of "product code =001, five-level classification = abnormal" the asset data records are filtered out by the adaptive condition, and do not participate in subsequent evaluation aggregation calculation, so that the validity of the data is improved.
It should be noted that the compression condition that can be adopted here may be a single compression condition, such as "product code = 001", or may be a composite compression condition, such as "product code =001, five-level classification = normal", or may be a mixture of a plurality of composite compression conditions.
And S109, performing aggregation calculation according to the aggregation calculation formula and the compressed dimension reduction data to generate an aggregation calculation result aiming at the target object.
After the compressed dimension reduction data is obtained, the aggregation calculation formula in the aggregation calculation protocol and the compressed dimension reduction data can be adopted to perform the aggregation calculation. Specifically, the values of the calculation attributes in the compressed dimension reduction data are input into an aggregation calculation formula, so as to obtain the corresponding aggregation calculation result.
In an embodiment, since there may be multiple aggregation calculation formulas in the aggregation calculation protocol, compression may be performed according to adaptive conditions in different aggregation calculation formulas when compressing dimension reduction data, so as to obtain compressed dimension reduction data corresponding to different aggregation calculation formulas.
In this embodiment, it is necessary to calculate, for each aggregation calculation formula, the compressed data to be aggregated related to the aggregation calculation formula to obtain an aggregation calculation result of the aggregation calculation formula, and further, accumulate the aggregation calculation results obtained by each aggregation calculation formula to obtain an aggregation calculation result of the target object.
It should be noted that, in this calculation manner, each aggregation calculation formula may have different compression conditions due to different adaptive conditions during aggregation, and thus the obtained compressed data may also have different compression conditions. Of course, if the aggregation calculation formulas all use the preset compression conditions uniformly, the same compressed data can be obtained.
According to the scheme provided by the embodiment of the specification, based on a multi-dimensional model established in advance, an aggregation calculation protocol is determined for data to be aggregated which needs to be calculated, calculation attributes which need to be used in calculation are dynamically obtained, dimension reduction is performed on the data to be aggregated, then the dimension reduction data are compressed, then the compressed dimension reduction data are used for carrying out aggregation calculation, invalid judgment and calculation are reduced, the calculation amount in aggregation is reduced, and efficient aggregation calculation of large-scale data is achieved.
In an embodiment, when determining the aggregation calculation protocol adopted for the target object, the determination of the aggregation calculation protocol may also be performed according to a type to which the target object belongs. For example, in an asset valuation scenario, configuration is performed in a pre-established underlying model, the type of each organization is determined according to the service content of each organization, and a valuation protocol adopted by a certain type is configured. Therefore, when the estimation needs to be carried out on a certain target mechanism, the type of the mechanism can be obtained, and the estimation protocol adopted by the target mechanism is determined according to the mechanism type, and in this way, the obtained estimation is generally the overall accurate estimation on the target mechanism, which is beneficial for each party to carry out accurate estimation on the assets of the target mechanism.
Further, in practical applications, one or more related objects (i.e. purchasing mechanisms) may enter the asset pool to be evaluated, and in this way, the type of the related object may be determined first, and then an aggregate computing protocol adapted to multiple related objects at the same time (for example, an aggregate computing protocol in a set of protocols adapted to multiple related objects at the same time) may be determined, so as to perform subsequent evaluation, thereby being closer to the practical needs of the user. In this case, the determined aggregation calculation protocol is often concentrated in a certain partial field in which the buyer is interested, so that the individualized evaluation of the assets of the target organization can be performed for each associated object, and the method is favorable for each buyer to perform the field evaluation based on the needs of each buyer.
Of course, in practical applications, it may also be determined and determined that the aggregation calculation protocol is simultaneously adapted to the target object and the associated object, so as to perform the aggregation calculation, which is not limited in this specification.
Correspondingly, an embodiment of the present specification further provides a data aggregation calculation apparatus, as shown in fig. 4, fig. 4 is a schematic structural diagram of the data aggregation calculation apparatus provided in the embodiment of the present specification, and includes:
the acquiring module 401 acquires a plurality of pieces of data to be aggregated of a target object, where the data to be aggregated includes a plurality of fixed attributes;
a determining module 403, configured to determine an aggregation calculation protocol adopted for a target object, where the aggregation calculation protocol includes at least one aggregation calculation formula;
a filtering module 405, configured to determine multiple calculation attributes required to be adopted by the aggregation calculation protocol, filter fixed attributes included in the data to be aggregated, and generate dimension reduction data only including the calculation attributes, where the calculation attributes are fixed attributes required to be adopted in the aggregation calculation formula;
the compression module 407 determines a compression condition, where the compression condition includes a part of the calculation attribute, compresses the remaining part of the calculation attribute in the dimension reduction data according to the compression condition, and generates compressed dimension reduction data;
and the aggregation calculation module 409 performs aggregation calculation according to the aggregation calculation formula and the compressed dimension reduction data to generate an aggregation calculation result for the target object.
Further, the apparatus further includes a writing module 411, which determines a fixed attribute included in any data to be aggregated, and writes the fixed attribute into the database with the target object as a primary key.
Further, the determining module 403 determines a type of the target object, and determines an aggregation calculation protocol adopted for the target object according to the type; or determining the type of the associated object of the data to be aggregated, acquiring the associated object of the data to be aggregated, and determining the aggregation calculation protocol adopted according to the target pair according to the type of the associated object.
Further, the compressing module 407 obtains an applicable condition of the aggregation calculation formula, and determines the applicable condition as a compression condition.
Further, when there are multiple aggregation calculation formulas, the aggregation calculation module 409 calculates, for any aggregation calculation formula, the compressed dimension reduction data related to the aggregation calculation formula to obtain an aggregation result of the aggregation calculation formula; aggregating respective aggregated results of the plurality of aggregated calculation formulas to generate an aggregated calculation result for the target object.
Embodiments of the present specification also provide a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aggregation calculation method of data shown in fig. 1 when executing the program.
Fig. 5 is a schematic diagram illustrating a more specific hardware structure of a computing device according to an embodiment of the present disclosure, where the computing device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called to be executed by the processor 1010.
The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present apparatus and other apparatuses. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
Embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the aggregation calculation method of data shown in fig. 1.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
From the above description of the embodiments, it is clear to those skilled in the art that the embodiments of the present disclosure can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the embodiments of the present specification may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments of the present specification.
The systems, methods, modules or units described in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the method embodiment, since it is substantially similar to the method embodiment, it is relatively simple to describe, and reference may be made to the partial description of the method embodiment for relevant points. The above-described method embodiments are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the functions of the modules may be implemented in one or more software and/or hardware when implementing the embodiments of the present specification. And part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The foregoing is only a specific embodiment of the embodiments of the present disclosure, and it should be noted that, for those skilled in the art, a plurality of modifications and decorations can be made without departing from the principle of the embodiments of the present disclosure, and these modifications and decorations should also be regarded as the protection scope of the embodiments of the present disclosure.
Claims (11)
1. A method of aggregate computation of data, comprising:
acquiring a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes;
determining an aggregation calculation protocol adopted by a target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula;
determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
determining a compression condition, wherein the compression condition comprises a part of calculation attributes, compressing the rest part of calculation attributes in the dimension reduction data according to the compression condition, and generating compressed dimension reduction data;
and performing aggregation calculation according to the aggregation calculation formula and the compressed dimension reduction data to generate an aggregation calculation result aiming at the target object.
2. The method of claim 1, prior to obtaining the plurality of pieces of data to be aggregated for the target object, the method further comprising:
and for any data to be aggregated, determining the fixed attributes contained in the data, and writing the data into a database with the target object as a primary key.
3. The method of claim 1, determining an aggregate computing protocol to employ for the target object, comprising:
determining the type of the target object, and determining an aggregation calculation protocol adopted by the target object according to the type; alternatively, the first and second electrodes may be,
determining the type of the associated object of the data to be aggregated, and determining the aggregation calculation protocol adopted according to the target pair according to the type of the associated object.
4. The method of claim 1, determining compression conditions, comprising:
and acquiring the applicable condition of the aggregation calculation formula, and determining the applicable condition as a compression condition.
5. The method of claim 4, when there are multiple aggregation calculation formulas, performing aggregation calculation according to the aggregation calculation formulas and the compressed dimension reduction data, and generating an aggregation calculation result for the target object, comprising:
calculating compressed dimension reduction data related to any aggregation calculation formula to obtain an aggregation result of the aggregation calculation formula;
aggregating the aggregated results of the plurality of aggregated calculation formulas to generate an aggregated calculation result for the target object.
6. An apparatus for aggregating computing data, comprising:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module acquires a plurality of pieces of data to be aggregated of a target object, and the data to be aggregated comprises a plurality of fixed attributes;
the determining module is used for determining an aggregation calculation protocol adopted by a target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula;
the filtering module is used for determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
the compression module is used for determining a compression condition, wherein the compression condition comprises a part of calculation attributes, compressing the rest calculation attributes in the dimensionality reduction data according to the compression condition and generating compressed dimensionality reduction data;
and the aggregation calculation module is used for performing aggregation calculation according to the aggregation calculation formula and the compressed dimension reduction data to generate an aggregation calculation result aiming at the target object.
7. The apparatus of claim 6, further comprising a writing module for determining, for any data to be aggregated, the fixed attribute contained in the data to be aggregated, and writing the data to a database with the target object as a primary key.
8. The apparatus of claim 6, the determination module to determine a type of the target object, determine an aggregate computing protocol employed for the target object according to the type; or determining the type of the associated object of the data to be aggregated, and determining the aggregation calculation protocol adopted by the target object according to the type of the associated object.
9. The apparatus of claim 6, the compression module to obtain an applicable condition of the aggregation calculation formula, and to determine the applicable condition as a compression condition.
10. The apparatus according to claim 9, wherein when there are multiple aggregation calculation formulas, the aggregation calculation module calculates compressed dimension reduction data associated with any aggregation calculation formula to obtain an aggregation result of the aggregation calculation formula; aggregating the aggregated results of the plurality of aggregated calculation formulas to generate an aggregated calculation result for the target object.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 5 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010302403.7A CN111459901B (en) | 2020-04-17 | 2020-04-17 | Data aggregation calculation method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010302403.7A CN111459901B (en) | 2020-04-17 | 2020-04-17 | Data aggregation calculation method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111459901A true CN111459901A (en) | 2020-07-28 |
| CN111459901B CN111459901B (en) | 2023-10-24 |
Family
ID=71685951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010302403.7A Active CN111459901B (en) | 2020-04-17 | 2020-04-17 | Data aggregation calculation method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459901B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050192941A1 (en) * | 2004-02-27 | 2005-09-01 | Stefan Biedenstein | Fast aggregation of compressed data using full table scans |
| US8606666B1 (en) * | 2007-01-31 | 2013-12-10 | Experian Information Solutions, Inc. | System and method for providing an aggregation tool |
| US20140372368A1 (en) * | 2013-06-18 | 2014-12-18 | Sap Ag | Removing Group-By Characteristics in Formula Exception Aggregation |
| US20150142773A1 (en) * | 2013-11-18 | 2015-05-21 | Tobias Mindnich | Optimizing Database Queries Having Hierarchy Filters |
| US20160085832A1 (en) * | 2014-09-24 | 2016-03-24 | Richard L Lam | System and method of analyzing data using bitmap techniques |
| CN106569750A (en) * | 2016-11-09 | 2017-04-19 | 郑州云海信息技术有限公司 | Data compression method and device |
-
2020
- 2020-04-17 CN CN202010302403.7A patent/CN111459901B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050192941A1 (en) * | 2004-02-27 | 2005-09-01 | Stefan Biedenstein | Fast aggregation of compressed data using full table scans |
| US8606666B1 (en) * | 2007-01-31 | 2013-12-10 | Experian Information Solutions, Inc. | System and method for providing an aggregation tool |
| US20140372368A1 (en) * | 2013-06-18 | 2014-12-18 | Sap Ag | Removing Group-By Characteristics in Formula Exception Aggregation |
| US20150142773A1 (en) * | 2013-11-18 | 2015-05-21 | Tobias Mindnich | Optimizing Database Queries Having Hierarchy Filters |
| US20160085832A1 (en) * | 2014-09-24 | 2016-03-24 | Richard L Lam | System and method of analyzing data using bitmap techniques |
| CN106569750A (en) * | 2016-11-09 | 2017-04-19 | 郑州云海信息技术有限公司 | Data compression method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111459901B (en) | 2023-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2018001195A1 (en) | Method and device for controlling data risk | |
| Ghysels et al. | Testing for cointegration with temporally aggregated and mixed‐frequency time series | |
| CN107451918B (en) | Asset data management method and device | |
| CN109118094B (en) | Enterprise service application system based on credit system | |
| CN106910129B (en) | Asset data processing method, client and server | |
| TWI705401B (en) | Multi-currency refund service processing method and device | |
| WO2021213154A1 (en) | Blockchain data processing method, system, terminal, and computer-readable storage medium | |
| CN110489418B (en) | Data aggregation method and system | |
| TWI725527B (en) | Insurance transaction processing method and device | |
| CN110874786B (en) | False transaction group identification method, device and computer readable medium | |
| CN112052385A (en) | Investment and financing project recommendation method and device, electronic equipment and readable storage medium | |
| CN105303447A (en) | Method and device for carrying out credit rating through network information | |
| CN113010791A (en) | Search result display processing method and device and computer readable storage medium | |
| US20180276299A1 (en) | Systems and methods for flexible streaming aggregation for electronic transaction analysis | |
| CN111459901B (en) | Data aggregation calculation method, device and equipment | |
| CN110838053A (en) | Financial business system, financial processing method and computer storage medium thereof | |
| TWM582615U (en) | Device for quantitative trading analysis | |
| CN112131465A (en) | Activity information matching method, device, medium and terminal equipment | |
| CN111506613A (en) | Method, system, device and equipment for querying incidence relation of data record | |
| US20220327634A1 (en) | Generating relevant attribute data for benchmark comparison | |
| CN109544322B (en) | Loan acceptance financial information processing method and device | |
| RU2699068C1 (en) | System for guaranteed return of goods in retail sales network | |
| CN116362879A (en) | Transaction data simulation method, device, computer equipment, storage medium and product | |
| TWI517064B (en) | SPU attribute aggregation method, device and system | |
| CN117132335A (en) | Method and device for determining resource transaction value in resource contract and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |