CN111459901B - Data aggregation calculation method, device and equipment - Google Patents
Data aggregation calculation method, device and equipment Download PDFInfo
- Publication number
- CN111459901B CN111459901B CN202010302403.7A CN202010302403A CN111459901B CN 111459901 B CN111459901 B CN 111459901B CN 202010302403 A CN202010302403 A CN 202010302403A CN 111459901 B CN111459901 B CN 111459901B
- Authority
- CN
- China
- Prior art keywords
- data
- aggregation
- calculation
- aggregation calculation
- target object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 194
- 230000002776 aggregation Effects 0.000 title claims abstract description 141
- 238000004220 aggregation Methods 0.000 title claims abstract description 141
- 230000009467 reduction Effects 0.000 claims abstract description 42
- 238000000034 method Methods 0.000 claims abstract description 24
- 238000007906 compression Methods 0.000 claims description 48
- 230000006835 compression Effects 0.000 claims description 48
- 238000001914 filtration Methods 0.000 claims description 8
- 230000004931 aggregating effect Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 11
- 239000000047 product Substances 0.000 description 10
- 238000003860 storage Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000006978 adaptation Effects 0.000 description 5
- 230000008520 organization Effects 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000006227 byproduct Substances 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A method, a device and equipment for data aggregation calculation are disclosed. And determining an aggregation calculation protocol according to the data to be aggregated, which is required to be calculated, dynamically acquiring calculation attributes required to be used in calculation, reducing the dimension of the data to be aggregated, further compressing the dimension reduction data, and then performing aggregation calculation by using the compressed dimension reduction data.
Description
Technical Field
Embodiments of the present disclosure relate to the field of information technologies, and in particular, to a method, an apparatus, and a device for computing aggregation of data.
Background
In many fields, it is often necessary to aggregate data to obtain corresponding results. However, in practical applications, a huge amount of data to be aggregated of a target object may occur. For example, in asset valuation of an organization, there may be millions of asset records in the name of an organization or group, requiring significant computation if aggregation is desired piece by piece.
Based on this, a more efficient solution for the aggregate computation is needed.
Disclosure of Invention
The embodiment of the application aims to provide an aggregation calculation scheme with higher efficiency.
In order to solve the technical problems, the embodiment of the application is realized as follows:
a method of aggregate computation of data, comprising:
acquiring a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes;
determining an aggregation calculation protocol adopted for a target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula;
determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
determining compression conditions, wherein the compression conditions comprise partial calculation attributes, compressing the rest calculation attributes in the dimension reduction data according to the compression conditions, and generating compressed dimension reduction data;
and performing aggregation calculation according to the aggregation calculation formula and the compressed reduced data to generate an aggregation calculation result aiming at the target object.
Correspondingly, the embodiment of the present disclosure further provides a data aggregation computing device, including:
the acquisition module acquires a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes;
the system comprises a determining module, a judging module and a judging module, wherein the determining module determines an aggregation calculation protocol adopted for a target object, and the aggregation calculation protocol comprises at least one aggregation calculation formula;
the filtering module is used for determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
the compression module is used for determining compression conditions, wherein the compression conditions comprise partial calculation attributes, and compressing the rest calculation attributes in the dimension reduction data according to the compression conditions to generate compressed dimension reduction data;
and the aggregation calculation module is used for carrying out aggregation calculation according to the aggregation calculation formula and the compressed reduced-dimension data to generate an aggregation calculation result aiming at the target object.
According to the scheme provided by the embodiment of the specification, based on the pre-established multidimensional model, an aggregation calculation protocol is determined for data to be aggregated, which is required to be calculated, calculation attributes required to be used in calculation are dynamically acquired, dimension reduction is performed on the data to be aggregated, further the dimension reduction data are compressed, and then the compressed dimension reduction data are used for aggregation calculation, so that invalid judgment and calculation are reduced, the calculation amount in aggregation is reduced, and efficient aggregation calculation of large-scale data is realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the disclosure.
Further, not all of the effects described above need be achieved in any of the embodiments of the present specification.
Drawings
In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present description, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
FIG. 1 is a schematic flow chart of a method for computing an aggregate of data according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of dimension reduction data according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram of compressed dimension reduction data according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a data aggregation computing device according to an embodiment of the present disclosure;
fig. 5 is a schematic diagram of an apparatus for configuring the method of the embodiments of the present specification.
Detailed Description
In order for those skilled in the art to better understand the technical solutions in the embodiments of the present specification, the technical solutions in the embodiments of the present specification will be described in detail below with reference to the drawings in the embodiments of the present specification, and it is apparent that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification shall fall within the scope of protection.
Aggregation computing on large-scale data is a scenario that is often faced at present. Such as transaction statistics, asset valuations, and the like. Aggregate computation refers to a computation that performs computation on a set of values and returns to a single computation, including computation of averages, sums, extrema, counts, and so forth.
Securitization of assets is a currently common practice, taking asset valuation as a specific example, and thus valuation of assets is also a necessary practice in financial activities, which often requires aggregating asset records for an organization, which in turn contain some invalid asset records. In practical applications, many organizations often have large assets, for example, for multi-organization billions of assets, where a certain organization may have up to millions of assets, and in such cases, if an efficient real-time aggregate valuation is to be achieved, a significant number of server parallel computations need to be provided, which is not practical and inefficient. Based on this, the embodiments of the present specification provide an efficient data aggregation calculation method.
The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings. As shown in fig. 1, fig. 1 is a flow chart of a method for computing aggregation of data according to an embodiment of the present disclosure, where the flow specifically includes the following steps:
s101, acquiring a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes.
The target objects are different based on implementation scenes. For example, under the transaction statistics of an e-commerce platform, the target object may be a merchant; in the context of asset valuation, the target object may be a business entity or group.
In the application scenario of asset estimation, the data to be aggregated may be asset records in a large-scale asset pool containing a plurality of institutions, where each piece of data to be aggregated has a corresponding target object. For example, a finance institution with loan qualification can issue various loan-forming property business data records by securitization supported by the property to which its own project belongs. In an asset pool, where there are often a large number of institutions involved, each of which has a corresponding securitized asset for sale, asset valuation may be required if there are other buyers who want to buy the asset of one institution. At this point, such purchasers with intent may be considered to be affiliates of the data to be aggregated, and it is apparent that there may be one or more affiliates.
Because the asset pool is pre-existing, in other words, before that, for any asset of each institution, an underlying model applied to asset characterization can be pre-constructed at the server side, and the underlying model can contain multiple dimensions (i.e. fixed attributes), so that the attributes and characteristics of one asset data record are characterized, and can be combined at will theoretically, so that any asset data to be evaluated can be described by the data record containing the fixed attributes. As shown in table 1, table 1 is a fixed dimension schematic table for characterizing an asset in dimensions provided in embodiments of the present disclosure.
By describing any asset through the dimensions, each piece of asset data to be evaluated can be endowed with a corresponding fixed attribute, the value under each fixed attribute is determined, and a piece of data to be aggregated, which contains a plurality of fixed attributes and the fixed attribute values thereof, is generated, and can also be called as data to be evaluated under the scene of asset evaluation.
It should be noted that, in this manner, a piece of data to be aggregated may include a considerable number of dimensions, for example, assuming that 30 dimensions are provided in a predetermined model, each piece of data may have 30 fixed dimensions, and values of some of the fixed dimensions may be invalid values (for example, default values, or blank values, etc.).
In other scenarios, such as in the context of transaction statistics, the fixed attribute may have other dimensions, e.g., in the context of transaction statistics, a transaction may include a fixed attribute such as "product code", "purchase object", "buyer age", "transaction amount", etc.
In practical applications, the fixed attributes contained in the data to be aggregated may be different from each other or only partially the same. For example, asset data 1 to be evaluated includes asset dimension 1 through asset dimension 10, while asset data 2 to be evaluated includes asset dimension 5 through asset dimension 20.
Furthermore, in practical application, for the sake of statistics, the target objects (or the target objects to which the target objects belong) included in one piece of data to be aggregated may be categorized and stored. For example, in a relational database, a target mechanism included in asset data to be evaluated is stored as a primary key. Thereby facilitating the overall valuation of all assets of a target institution by other users.
Based on the foregoing, any of the affiliates (i.e., buyers, either individual or institution users) may obtain a plurality of pieces of asset data to be evaluated for the target institution from the pool of assets based on the identity of the target institution. In this process, the multiple pieces of asset data to be evaluated obtained back can be further screened based on a certain fixed asset attribute.
S103, determining an aggregation calculation protocol adopted for the target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula.
The bottom layer model of the server side can also be provided with a plurality of calculation formulas for calculating a piece of data to be aggregated in advance, and an aggregation calculation formula adapted by a certain target mechanism or a certain target mechanism is provided, so that a plurality of predefined aggregation calculation protocols are formed. For example, in the scenario of asset valuation, which may be referred to as a valuation protocol, as shown in Table 2, table 2 is a schematic table of an aggregate computing protocol as set forth in the embodiments herein.
PV, P, I, F, R and other characters in the table, etc. each characterize a certain fixed attribute in the data to be aggregated. In an aggregate computing protocol, at least one aggregate computing formula for computing should be included, and of course, a plurality of aggregate computing formulas may be included.
When the fixed attribute and the value contained in the aggregated data accord with the aggregation calculation formula in the table, the corresponding aggregation calculation formula can be used for carrying out aggregation calculation on the aggregated data.
S105, determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula.
The calculation attribute refers to a fixed attribute required to be adopted in an aggregate calculation formula. Taking equation 2 in table 2 as an example, the calculation attributes included therein are "PV", "P", "V", "I".
In each aggregate calculation formula, it is necessary to include calculation attributes for the calculation formula (i.e., fixed attributes for calculating the formula), and it is easy to understand that for a calculation formula, it generally includes a limited number of calculation attributes, the number of calculation attributes being much smaller than the fixed attributes.
For example, in one model, there may be 50 full-scale fixed production attributes, while the number of calculated attributes in a particular aggregate calculation formula may not exceed 5. So that for each aggregate calculation formula there will be a set of attributes corresponding to the calculation attributes that the aggregate calculation formula needs to employ.
When a plurality of aggregation calculation formulas exist in the aggregation calculation protocol, the calculation attributes required to be adopted are the union of attribute sets corresponding to the aggregation calculation formulas.
Further, dimension reduction can be performed on the data to be aggregated according to the determined multiple computing attributes, and invalid fixed asset dimensions (i.e. dimensions which are not adopted in the aggregation computing protocol) are filtered, so that dimension reduction data only comprising the computing attributes is generated.
As described above, the fixed attributes contained in the data to be aggregated may be different from each other or only partially the same, so that the dimensions of the data to be aggregated can be unified by the aforementioned dimension reduction method, thereby realizing the unification of the dimensions of the data.
S107, determining compression conditions, wherein the compression conditions comprise partial calculation attributes, compressing the rest calculation attributes in the dimension reduction data according to the compression conditions, and generating compressed dimension reduction data.
As previously mentioned, each data to be aggregated has now been filtered out of several dimensions and unification in data dimensions is achieved, but in practice the number is not reduced. As shown in fig. 2, fig. 2 is a schematic diagram of dimension reduction data according to an embodiment of the present disclosure.
Based on this, the reduced data can be compressed according to an appropriate compression condition. For example, in a relational database, the data may be compressed using group by in a structured query language (Structured Query Language, SQL).
For example, compression conditions are used for the data in fig. 2: sql "select sum (principal), sum (interest), sum (penalty), sum (cost) from olap_table window target organization= 'XXA' group by product code, five-level classification, repayment mode", data compression is performed, thereby obtaining compressed dimension reduction data as shown in fig. 3. The part of the calculation attributes contained in the method are the product codes, the five-stage classification and the repayment modes, and the rest of the calculation attributes are the principal, interest, penalty and expense.
Based on the compression mode, the compression of a plurality of pieces of data with the same value of calculation attribute into one piece of data record is realized, although in the schematic diagram, only 4 pieces of records are compressed into two pieces of data. However, in practical application, since the number of asset records having the same calculation attribute value in the same target institution may be quite large, in practical application, millions of assets of a certain institution may be compressed to hundreds, and the compression rate is quite large.
In the transaction statistics scenario, when a user of a certain product in each age group needs to count the transaction of the user in each time group, the corresponding compression is needed based on the time period and the age.
In practical application, which compression conditions are specifically needed to be selected for compression, in one embodiment, the compression conditions can be customized in advance according to the selection of a user, in another embodiment, the applicable conditions needed by the aggregation calculation formula in the aggregation calculation protocol can be further used for aggregating the rest calculation attributes in the data to be aggregated according to the calculation asset attributes contained in the applicable conditions, so that in this way, the aggregated data is the compression data corresponding to the aggregation calculation formula. Namely, the adaptation condition of the aggregation calculation formula is taken as the compression condition,
for example, for calculation formula 1 in table 2, the adaptation condition is "product code=001, five-level classification=normal", and then the data records of "product code=001, five-level classification=normal" in the calculated asset attribute may be aggregated, thereby obtaining the compressed data adapted to calculation formula 1.
In other words, the "determination of compression conditions" herein may require determination of not only the calculation attribute to be employed at the time of compression, but also the value of the calculation attribute. For example, in this case, if "product code=001, five-level classification=normal" is not included in the calculated attribute of one data record, the subsequent compression is not participated.
In this way, some invalid data can be filtered out, for example, in the estimation scene, for those asset data records with "product code=001 and five-level classification=abnormal" will be filtered out by the adaptation condition, and will not participate in subsequent estimation aggregation calculation, thus improving the effectiveness of the data.
It should be noted that the compression condition may be a single compression condition, for example, "product code=001", or a composite compression condition, for example, "product code=001, five-stage classification=normal", or a mixture of a plurality of composite compression conditions.
And S109, performing aggregation calculation according to the aggregation calculation formula and the compressed reduced data, and generating an aggregation calculation result aiming at the target object.
After the compressed dimensionality reduction data is obtained, the aggregation calculation formula in the aggregation calculation protocol and the compressed dimensionality reduction data can be adopted for aggregation calculation. Specifically, the values of all the calculation attributes in the compressed dimension reduction data are input into an aggregation calculation formula, so that a corresponding aggregation calculation result is obtained.
In one embodiment, since there may be multiple aggregation calculation formulas in the aggregation calculation protocol, when the dimension reduction data is compressed, the dimension reduction data corresponding to the compressed different aggregation calculation formulas may be obtained by compressing the dimension reduction data according to the adaptation conditions in the different aggregation calculation formulas.
In this embodiment, the compressed data to be aggregated related to each aggregation calculation formula needs to be calculated for each aggregation calculation formula to obtain an aggregation calculation result of the aggregation calculation formula, and further, the aggregation calculation results obtained by each aggregation calculation formula are accumulated, so as to obtain an aggregation calculation result of the target object.
It should be noted that in this calculation mode, each aggregation calculation formula may be different in compression condition due to different adaptation conditions during aggregation, and thus the obtained compressed data may be different. Of course, if preset compression conditions are uniformly adopted by all the aggregation calculation formulas, the same compression data can be obtained.
According to the scheme provided by the embodiment of the specification, based on the pre-established multidimensional model, an aggregation calculation protocol is determined for data to be aggregated, which needs to be calculated, calculation attributes needed to be used in calculation are dynamically acquired, dimension reduction is performed on the data to be aggregated, further the dimension reduction data are compressed, and then the compressed dimension reduction data are used for aggregation calculation, so that invalid judgment and calculation are reduced, the calculation amount in aggregation is reduced, and efficient aggregation calculation of large-scale data is realized.
In one embodiment, when determining the aggregate computing protocol adopted for the target object, the determining of the aggregate computing protocol may also be performed according to the type to which the target object belongs. For example, in an asset valuation scenario, configuration is performed in a pre-established underlying model, the type of each institution is determined from the business content of each institution, and the valuation protocol employed by a certain type is configured. Thus, whenever an estimate is required for a particular target institution, the institution type may be obtained, and the estimation protocol employed by the target institution may be determined based on the institution type, in which way the obtained estimate is typically an overall accurate estimate of the target institution, facilitating accurate estimates by parties for the asset of the target institution.
Further, in practical applications, one or more associated objects (i.e. purchasing institutions) may be needed to be evaluated in the asset pool, in this manner, the type of the associated object may be determined first, and then an aggregate computing protocol adapted to multiple associated objects at the same time (for example, a certain aggregate computing protocol in a protocol set adapted to multiple associated objects at the same time) may be determined, so as to perform subsequent evaluation, thereby being closer to the practical needs of the user. In this case, the determined aggregate computing protocol is often focused on a portion of the area of interest to the buyer, so that the asset of the target institution can be personalized for each associated object to perform the tiled evaluation, which is beneficial to the buyer to perform the territorial evaluation based on the respective needs.
Of course, in practical application, the type of the target object and the associated object may be determined, and an aggregation calculation protocol adapted to the target object and the associated object at the same time may be determined to perform the aggregation calculation, which is not limited in this specification.
Correspondingly, the embodiment of the present disclosure further provides a data aggregation computing device, as shown in fig. 4, and fig. 4 is a schematic structural diagram of the data aggregation computing device provided in the embodiment of the present disclosure, including:
the acquiring module 401 acquires a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated contains a plurality of fixed attributes;
a determining module 403, configured to determine an aggregate computing protocol adopted for the target object, where the aggregate computing protocol includes at least one aggregate computing formula;
the filtering module 405 determines a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filters fixed attributes contained in the data to be aggregated, and generates dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
the compression module 407 determines a compression condition, wherein the compression condition comprises a part of calculation attributes, and compresses the rest calculation attributes in the dimension reduction data according to the compression condition to generate compressed dimension reduction data;
and an aggregation calculation module 409, which performs aggregation calculation according to the aggregation calculation formula and the compressed dimension-reduced data, to generate an aggregation calculation result for the target object.
Further, the apparatus further includes a writing module 411, for any data to be aggregated, determining a fixed attribute contained in the data to be aggregated, and writing the fixed attribute into a database using the target object as a primary key.
Further, the determining module 403 determines a type of the target object, and determines an aggregate computing protocol adopted for the target object according to the type; or determining the type of the associated object of the data to be aggregated, acquiring the associated object of the data to be aggregated, and determining an aggregation calculation protocol adopted according to the target pair according to the type of the associated object.
Further, the compression module 407 obtains the applicable condition of the aggregation calculation formula, and determines the applicable condition as a compression condition.
Further, when there are multiple aggregation calculation formulas, the aggregation calculation module 409 calculates the compressed dimension reduction data related to the aggregation calculation formula for any aggregation calculation formula, so as to obtain an aggregation result of the aggregation calculation formula; and aggregating the aggregation results of the aggregation calculation formulas to generate an aggregation calculation result aiming at the target object.
The embodiments of the present disclosure also provide a computer device, which at least includes a memory, a processor, and a computer program stored on the memory and capable of running on the processor, wherein the processor implements the method for aggregating and computing data shown in fig. 1 when executing the program.
FIG. 5 illustrates a more specific hardware architecture diagram of a computing device provided by embodiments of the present description, which may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 implement communication connections therebetween within the device via a bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit ), microprocessor, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc. for executing relevant programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory ), static storage device, dynamic storage device, or the like. Memory 1020 may store an operating system and other application programs, and when the embodiments of the present specification are implemented in software or firmware, the associated program code is stored in memory 1020 and executed by processor 1010.
The input/output interface 1030 is used to connect with an input/output module for inputting and outputting information. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.
Communication interface 1040 is used to connect communication modules (not shown) to enable communication interactions of the present device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).
Bus 1050 includes a path for transferring information between components of the device (e.g., processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).
It should be noted that although the above-described device only shows processor 1010, memory 1020, input/output interface 1030, communication interface 1040, and bus 1050, in an implementation, the device may include other components necessary to achieve proper operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the aggregate computing method of data shown in fig. 1.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
From the foregoing description of embodiments, it will be apparent to those skilled in the art that the present embodiments may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions of the embodiments of the present specification may be embodied in essence or what contributes to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments of the present specification.
The system, method, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the method embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points. The above-described method embodiments are merely illustrative, in that the modules illustrated as separate components may or may not be physically separate, and the functions of the modules may be implemented in the same piece or pieces of software and/or hardware when implementing the embodiments of the present disclosure. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present application without undue burden.
The foregoing is merely a specific implementation of the embodiments of this disclosure, and it should be noted that, for a person skilled in the art, several improvements and modifications may be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered as protective scope of the embodiments of this disclosure.
Claims (11)
1. A method of aggregate computation of data, comprising:
acquiring a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes; the target object is a financial institution, the data to be aggregated is an asset record of the financial institution, and the fixed attribute is an asset dimension owned by the asset record;
determining an aggregation calculation protocol adopted for a target object, wherein the aggregation calculation protocol comprises at least one aggregation calculation formula; the aggregation calculation protocol is an estimation protocol;
determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
determining compression conditions, wherein the compression conditions comprise partial calculation attributes, compressing the rest calculation attributes in the dimension reduction data according to the compression conditions, and generating compressed dimension reduction data;
and performing aggregation calculation according to the aggregation calculation formula and the compressed reduced data to generate an aggregation calculation result aiming at the target object, wherein the aggregation calculation result is used for representing the asset valuation of the financial institution.
2. The method of claim 1, prior to acquiring the plurality of pieces of data to be aggregated for the target object, the method further comprising:
and determining the fixed attribute contained in any data to be aggregated, and writing the fixed attribute into a database taking the target object as a main key.
3. The method of claim 1, determining an aggregate computing protocol employed for the target object, comprising:
determining the type of the target object, and determining an aggregation calculation protocol adopted for the target object according to the type; or,
and determining the type of the associated object of the data to be aggregated, and determining an aggregation calculation protocol adopted for the target object according to the type of the associated object.
4. The method of claim 1, determining compression conditions comprising:
and acquiring the applicable conditions of the aggregation calculation formula, and determining the applicable conditions as compression conditions.
5. The method of claim 4, when there are a plurality of aggregate formulas, performing aggregate calculation according to the aggregate formulas and the compressed reduced data, generating an aggregate calculation result for the target object, comprising:
aiming at any aggregation calculation formula, calculating the compressed dimension reduction data related to the aggregation calculation formula to obtain an aggregation result of the aggregation calculation formula;
and aggregating the aggregation results of the aggregation calculation formulas to generate an aggregation calculation result aiming at the target object.
6. An aggregate computing device of data, comprising:
the acquisition module acquires a plurality of pieces of data to be aggregated of a target object, wherein the data to be aggregated comprises a plurality of fixed attributes; the target object is a financial institution, the data to be aggregated is an asset record of the financial institution, and the fixed attribute is an asset dimension owned by the asset record;
the system comprises a determining module, a judging module and a judging module, wherein the determining module determines an aggregation calculation protocol adopted for a target object, and the aggregation calculation protocol comprises at least one aggregation calculation formula; the aggregation calculation protocol is an estimation protocol;
the filtering module is used for determining a plurality of calculation attributes required to be adopted by the aggregation calculation protocol, filtering fixed attributes contained in the data to be aggregated, and generating dimension reduction data only containing the calculation attributes, wherein the calculation attributes are the fixed attributes required to be adopted in the aggregation calculation formula;
the compression module is used for determining compression conditions, wherein the compression conditions comprise partial calculation attributes, and compressing the rest calculation attributes in the dimension reduction data according to the compression conditions to generate compressed dimension reduction data;
and the aggregation calculation module is used for carrying out aggregation calculation according to the aggregation calculation formula and the compressed reduced-dimension data to generate an aggregation calculation result aiming at the target object, wherein the aggregation calculation result is used for representing the asset estimation of the financial institution.
7. The apparatus of claim 6, further comprising a writing module configured to determine, for any data to be aggregated, a fixed attribute contained therein, and write the fixed attribute into a database having the target object as a primary key.
8. The apparatus of claim 6, the determination module to determine a type of the target object, and to determine an aggregate computing protocol employed for the target object based on the type; or determining the type of the associated object of the data to be aggregated, and determining an aggregation calculation protocol adopted for the target object according to the type of the associated object.
9. The apparatus of claim 6, the compression module to obtain applicable conditions of the aggregate calculation formula, the applicable conditions to determine as compression conditions.
10. The apparatus of claim 9, wherein when there are a plurality of aggregation calculation formulas, the aggregation calculation module calculates a compressed dimension reduction data related to the aggregation calculation formula for any aggregation calculation formula to obtain an aggregation result of the aggregation calculation formula; and aggregating the aggregation results of the aggregation calculation formulas to generate an aggregation calculation result aiming at the target object.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 5 when the program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010302403.7A CN111459901B (en) | 2020-04-17 | 2020-04-17 | Data aggregation calculation method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010302403.7A CN111459901B (en) | 2020-04-17 | 2020-04-17 | Data aggregation calculation method, device and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111459901A CN111459901A (en) | 2020-07-28 |
| CN111459901B true CN111459901B (en) | 2023-10-24 |
Family
ID=71685951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010302403.7A Active CN111459901B (en) | 2020-04-17 | 2020-04-17 | Data aggregation calculation method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459901B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8606666B1 (en) * | 2007-01-31 | 2013-12-10 | Experian Information Solutions, Inc. | System and method for providing an aggregation tool |
| CN106569750A (en) * | 2016-11-09 | 2017-04-19 | 郑州云海信息技术有限公司 | Data compression method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7263520B2 (en) * | 2004-02-27 | 2007-08-28 | Sap Ag | Fast aggregation of compressed data using full table scans |
| US9378264B2 (en) * | 2013-06-18 | 2016-06-28 | Sap Se | Removing group-by characteristics in formula exception aggregation |
| US9767151B2 (en) * | 2013-11-18 | 2017-09-19 | Sap Se | Optimizing database queries having hierarchy filters |
| US20160085832A1 (en) * | 2014-09-24 | 2016-03-24 | Richard L Lam | System and method of analyzing data using bitmap techniques |
-
2020
- 2020-04-17 CN CN202010302403.7A patent/CN111459901B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8606666B1 (en) * | 2007-01-31 | 2013-12-10 | Experian Information Solutions, Inc. | System and method for providing an aggregation tool |
| CN106569750A (en) * | 2016-11-09 | 2017-04-19 | 郑州云海信息技术有限公司 | Data compression method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111459901A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI673666B (en) | Method and device for data risk control | |
| CN109102151B (en) | Suspicious group identification method and device | |
| US20150278813A1 (en) | Determining a temporary transaction limit | |
| US20210312460A1 (en) | Method and device for identifying a risk merchant | |
| CN106960382A (en) | A kind of method of data processing, device and server | |
| WO2018219285A1 (en) | Data object display method and device | |
| CN102129631B (en) | Method, equipment and system for SPU attribute aggregation | |
| CN107451918B (en) | Asset data management method and device | |
| WO2022007592A1 (en) | Multidimensional data analysis method, apparatus, and system | |
| CN110489418B (en) | Data aggregation method and system | |
| WO2020042764A1 (en) | Amount settlement system and method | |
| TWI705401B (en) | Multi-currency refund service processing method and device | |
| CN111125266A (en) | Data processing method, device, equipment and storage medium | |
| CN110750530A (en) | Service system and data checking method thereof | |
| CN105303447A (en) | Method and device for carrying out credit rating through network information | |
| CN107704941A (en) | A kind of method and device for showing goods review | |
| CN110874786B (en) | False transaction group identification method, device and computer readable medium | |
| TW202032483A (en) | Insurance transaction processing method and device | |
| CN113010791A (en) | Search result display processing method and device and computer readable storage medium | |
| CN111459901B (en) | Data aggregation calculation method, device and equipment | |
| CN112182071A (en) | Data association relation mining method and device, electronic equipment and storage medium | |
| US20140114816A1 (en) | Multidimensional measurement basis for accounting systems | |
| US20180276299A1 (en) | Systems and methods for flexible streaming aggregation for electronic transaction analysis | |
| CN114387085B (en) | Method, device, computer equipment and storage medium for processing stream data | |
| US9230022B1 (en) | Customizable result sets for application program interfaces |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |