CN111522918A - Data aggregation method and device, electronic equipment and computer readable storage medium - Google Patents
Data aggregation method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111522918A CN111522918A CN202010331096.5A CN202010331096A CN111522918A CN 111522918 A CN111522918 A CN 111522918A CN 202010331096 A CN202010331096 A CN 202010331096A CN 111522918 A CN111522918 A CN 111522918A
- Authority
- CN
- China
- Prior art keywords
- data
- search engine
- full
- target
- text search
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/183—Tabulation, i.e. one-dimensional positioning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data aggregation method, a data aggregation device, electronic equipment and a computer readable storage medium, which are applied to the technical field of data warehouses, wherein when a data source is monitored to change, first target data are extracted according to a preset rule, data in a dimension table stored by a full text search engine are updated, in other words, the changed data are only aggregated, the data are prevented from being aggregated integrally, the data processing amount is reduced, the data can be aggregated in real time, the efficiency of acquiring the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and the information hysteresis of report generation or BI analysis is avoided.
Description
Technical Field
The present application relates to the field of data warehouse technologies, and in particular, to a data aggregation method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
As enterprise traffic grows, it has become difficult for a single database to store the explosion of data as traffic grows, and enterprises typically store vast amounts of business data through distributed database clusters. Based on the service requirement, service managers often need to gather data scattered in different databases, and then make or analyze reports based on the gathered data.
According to the current data aggregation method, service management personnel can only generate reports or perform BI analysis based on the data stored in history, or firstly perform overall aggregation of the data of each database, and then generate reports or perform BI analysis. However, according to the current data aggregation method, a report is generated or BI analysis is performed based on the data stored in history, and the report cannot be generated or BI analysis cannot be performed based on the latest data, so that the problems of analysis lag and poor accuracy exist; the data of each database are integrally gathered, and then a report form is generated or BI analysis is performed, so that the integral gathering of the data of each database is time-consuming, and the problem of low efficiency exists.
Disclosure of Invention
The application provides a data aggregation method, a data aggregation device, electronic equipment and a computer-readable storage medium, which can aggregate changed data in real time, improve the efficiency of acquiring latest data, and avoid information hysteresis of analysis based on historical data, and the technical scheme adopted by the application is as follows:
in a first aspect, a data aggregation method is provided, the method comprising,
when monitoring that the data source of the target database changes, extracting first target data according to a preset rule;
and updating the data in the dimension table stored by the full-text search engine based on the extracted first target data.
Specifically, when it is monitored that a data source of the target database changes, extracting first target data according to a predetermined rule includes:
determining a dimension model corresponding to each changed data based on a routing rule, wherein the dimension model comprises master table information, slave table information, master key information and associated parameter information;
and acquiring the associated parameters of the corresponding master table and/or slave table from the changed data in the data source respectively based on the master key information of all the corresponding dimension models to serve as first target data.
Specifically, updating data in a dimension table stored by a full-text search engine based on the extracted first target data comprises the following steps:
and respectively updating corresponding data in the dimension table on the full-text search engine based on the type of the data corresponding to the first target data, wherein the type comprises master table data and slave table data.
Further, the method comprises:
extracting second target data from the target database based on all the dimension models;
and generating a dimension table based on the second target data and storing the dimension table to the full-text search engine.
Further, the method further comprises: receiving a report generation request and/or a BI analysis request;
generating a report based on the updated dimension table and/or performing BI analysis.
Wherein the database comprises a relational database and/or a non-relational database.
Wherein, the full-text search engine is a Solr full-text search engine or an elastic search full-text search engine.
In a second aspect, there is provided a data aggregation device, comprising,
the monitoring module is used for extracting first target data according to a preset rule when monitoring that a data source of the target database changes;
and the updating module is used for updating the data in the dimension table stored by the full-text search engine based on the extracted first target data.
Specifically, the monitoring module includes:
the determining unit is used for determining a dimension model corresponding to each changed data based on the routing rule, wherein the dimension model comprises master table information, slave table information, master key information and associated parameter information;
and the acquisition unit is used for acquiring the association parameters of the corresponding master table and/or slave table from the changed data in the data source as first target data based on the master key information of all the corresponding dimension models.
Specifically, the updating module is specifically configured to update corresponding data in the dimension table on the full-text search engine based on a type of data corresponding to the first target data, where the type includes master table data and slave table data.
Further, the apparatus further comprises:
the extraction module is used for extracting second target data from the target database based on all the dimension models;
and the storage module is used for generating a dimension table based on the second target data and storing the dimension table to the full-text search engine.
Further, the apparatus further comprises:
the device also includes:
the receiving module is used for receiving a report generation request and/or a BI analysis request;
and the analysis module is used for generating a report and/or performing BI analysis based on the updated dimension table.
Wherein the database comprises a relational database and/or a non-relational database.
The full-text search engine is a Solr full-text search engine or an elastic search full-text search engine.
In a third aspect, an electronic device is provided, which includes:
one or more processors;
a memory;
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: the data aggregation method shown in the first aspect is performed.
In a fourth aspect, a computer-readable storage medium is provided, which is used for storing computer instructions, which when run on a computer, make the computer perform the data aggregation method shown in the first aspect.
Compared with the prior art that the latest data acquisition is time-consuming and low in efficiency, the method and the device for data aggregation, the electronic equipment and the computer-readable storage medium extract the first target data according to the preset rule when the change of the data source of the target database is monitored, and update the data in the dimension table stored by the full-text search engine based on the extracted first target data. That is, when the change of the data source is monitored, the first target data is extracted according to the preset rule, and the data in the dimension table stored by the full-text search engine is updated, in other words, only the changed data is converged, so that the data is prevented from being converged integrally, the data processing amount is reduced, the real-time convergence of the data can be realized, the efficiency of obtaining the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and the information hysteresis of report generation or BI analysis is avoided.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic flowchart of a data aggregation method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a data aggregation device according to an embodiment of the present application;
FIG. 3 is a schematic structural diagram of another data aggregation apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.
As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
An embodiment of the present application provides a data aggregation method, as shown in fig. 1, the method may include the following steps:
step S101, when monitoring that a data source of a target database changes, extracting first target data according to a preset rule;
specifically, when the change of a data source in a target database is monitored through a corresponding monitoring method, first target data are extracted from the changed data according to a preset rule; the monitoring can be realized by monitoring database logs, namely, data of generating insert, update, delete and other operations are identified by monitoring and analyzing binlog of the database; the target database may be one database or a plurality of databases, wherein the plurality of databases may be different types of databases, and the monitoring rules of the databases may be different; the first target data may be all data that has changed, or may be partial data or partial parameters of the data that has changed, that is, only partial data of the data that has changed is extracted.
And step S102, updating the data in the dimension table stored by the full-text search engine based on the extracted first target data.
Specifically, the full-text search engine stores historical data of corresponding databases; wherein the respective databases may be distributed deployed; the full text search engine is a mainstream search engine widely applied at present, and the working principle of the full text search engine is that a computer indexing program establishes an index for each word by scanning each word in an article to indicate the occurrence frequency and position of the word in the article, when a user inquires, the retrieval program searches according to the established index in advance and feeds back the searched result to the retrieval mode of the user;
specifically, updating is performed based on the relationship between the first target data and the data in the dimension table, and if the first target data replaces a certain attribute parameter of the historical data, the corresponding data in the dimension table may be updated according to the mapping relationship between the corresponding first target data and the corresponding data in the dimension table; if the first target data is added with some data on the basis of the history data, the first target data can be added in the dimension table.
Compared with the prior art that the time is very long and the efficiency is low when the latest data are acquired, the method for acquiring the data in the dimension table stored by the full-text search engine is characterized in that the first target data are extracted according to the preset rule when the change of the data source of the target database is monitored, and the data in the dimension table stored by the full-text search engine are updated based on the extracted first target data. That is, according to the embodiment of the present application, when it is monitored that a data source changes, the first target data is extracted according to a predetermined rule, and data in a dimension table stored in a full-text search engine is updated, in other words, only the changed data is aggregated, aggregation of the entire data is avoided, data processing amount is reduced, and real-time aggregation of the data can be achieved, so that efficiency of obtaining the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and information hysteresis of report generation or BI analysis is avoided.
The embodiment of the present application provides a possible implementation manner, and specifically, step S101 includes:
step S1011 (not shown in the figure), determining a dimension model corresponding to each changed data based on the routing rule, where the dimension model includes master table information, slave table information, master key information, and associated parameter information;
specifically, when it is monitored that a data source of the target database changes, determining, by using a corresponding routing rule, dimension models corresponding to changed data, for example, the changed data are data a, data B, and data C, and the corresponding dimension models are a dimension model a, a dimension model B, and a dimension model C, where each dimension model defines information including, but not limited to, the following information: the data information of the source data which is extracted to change is different, and the main table information, the sub table information, the main key information and the associated parameter information which are defined by each dimension model can be different; the master table is only one, the slave tables can be multiple, both the slave tables and the master table contain master key information, a relationship between the master table and the slave tables can be established through the master key, the association parameters define data extracted from the master table or the slave tables, illustratively, the master key information can be identity number information, and when the information of a certain client is monitored to be changed, corresponding information can be obtained from the master table and/or the slave tables through the master key information.
Step S1012 (not shown in the figure), obtaining the associated parameters of the corresponding master table and/or slave table from the changed data in the data source based on the primary key information of all the corresponding dimension models, respectively, as the first target data.
Specifically, the changed data may be data in a master table and/or a slave table, and the associated parameters in the corresponding master table and/or slave table may be acquired as the first target data through the master key information.
With the embodiment of the application, the problem of how to extract the first target data according to the predetermined rule is solved, that is, the problem of how to obtain the first target data when the data source in the database is monitored to be changed is solved.
The embodiment of the present application provides a possible implementation manner, and specifically, step S102 includes:
step S1021 (not shown in the figure), based on the type of the data corresponding to the first target data, respectively update the corresponding data in the dimension table on the full-text search engine, where the type includes the master table data and the slave table data.
Specifically, according to different data types corresponding to the first target data, corresponding data in a dimension table on a full-text search engine are respectively updated; wherein, the data type can be master table data and slave table data; wherein the first target data may comprise master table data and/or slave table data.
For example, the first target data includes both master table data and slave table data, and for the master table data of the first target data, the corresponding master table data and slave table data on the full-text search engine need to be updated; and for the slave table data of the first target data, only updating corresponding slave table data on the full-text search engine, wherein the corresponding master table and slave table on the full-text search engine can be obtained through primary key information query.
According to the embodiment of the application, the problem of how to update the corresponding data in the dimension table on the full-text search engine according to the first target data is solved.
The embodiment of the present application provides a possible implementation manner, and further, the method further includes:
step S103 (not shown in the figure), extracting second target data from the target database based on all the dimension models;
step S104 (not shown in the figure), a dimension table is generated based on the second target data and stored to the full-text search engine.
Specifically, when the historical data is extracted integrally, second target data is extracted from the target database based on all the dimension models, and a dimension table is generated based on the second target data and stored in the full-text search engine.
According to the embodiment of the application, the problem of how to realize the integral aggregation of the historical data is solved.
The embodiment of the present application provides a possible implementation manner, and further, the method further includes:
step S105 (not shown in the figure), receiving a report generation request and/or a BI analysis request;
step S106 (not shown), generating report and/or performing BI analysis based on the updated dimension table.
Specifically, when a report generation request and/or a BI (business intelligence) analysis request is received, a report is generated and/or BI analysis is performed based on the updated dimension table; the report generation request and/or the BI analysis request may be implemented by the relevant service personnel by triggering the corresponding virtual key.
According to the embodiment of the application, the report can be generated in real time and/or the BI can be analyzed in real time.
The embodiments of the present application provide a possible implementation manner, where the target database includes a relational database and/or a non-relational database.
Specifically, the target databases may be multiple, and the multiple databases may include a relational database and a non-relational database, where monitoring rules corresponding to the relational database and the non-relational database are different, and when a change in a data source of the corresponding database is monitored, data is extracted.
For the embodiment of the application, the data aggregation of the relational database and the non-relational database is realized.
The embodiment of the application provides a possible implementation manner, wherein the full-text search engine is a Solr full-text search engine or an elastic search full-text search engine.
Specifically, the full-text search engine may be a Solr full-text search engine or an elastic search full-text search engine, or may be another search engine capable of implementing the functions of the present application; the Solr is a high-performance full-text search engine, is developed by adopting Java, is based on a Lucene full-text search server, is expanded, provides richer query languages than Lucene, realizes configurability and expandability, optimizes the query performance, provides a perfect function management interface and is a very excellent full-text search engine; the Elasticisearch is a RESTful search engine constructed based on an Apache lucene library, provides a distributed full-text search engine with multi-tenant capability, and has an HTTPWeb interface (REST) and a non-architecture JSON document. It is worth noting that compared with the prior art that the BI analysis is performed by using a local storage, the BI analysis is performed based on the elastic search full-text search engine, the analyzed data volume can be expanded transversely, the BI analysis is not limited by the analyzed data volume, the problem that the data volume for performing the BI analysis is limited in the prior art is solved, and moreover, the data query speed based on the elastic search is very high, so that the efficiency of generating a report and performing the BI analysis can be improved.
For the embodiment of the application, the dimension table obtained by aggregation is stored in a Solr full-text search engine or an elastic search full-text search engine, so that the problems of subsequent report generation and BI analysis are solved.
Fig. 2 is a data aggregation apparatus provided in an embodiment of the present application, where the apparatus 20 includes: a monitoring module 201 and an updating module 202, wherein,
the monitoring module 201 is configured to extract first target data according to a predetermined rule when a change in a data source of a target database is monitored;
and the updating module 202 is used for updating the data in the dimension table stored by the full-text search engine based on the extracted first target data.
Compared with the prior art that time is consumed for obtaining the latest data and efficiency is low, the data aggregation device extracts the first target data according to the preset rule when the change of the data source of the target database is monitored, and updates the data in the dimension table stored by the full-text search engine based on the extracted first target data. That is, according to the embodiment of the present application, when it is monitored that a data source changes, the first target data is extracted according to a predetermined rule, and data in a dimension table stored in a full-text search engine is updated, in other words, only the changed data is aggregated, aggregation of the entire data is avoided, data processing amount is reduced, and real-time aggregation of the data can be achieved, so that efficiency of obtaining the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and information hysteresis of report generation or BI analysis is avoided.
The data aggregation device of this embodiment may execute a data aggregation method provided in the above embodiments of this application, and the implementation principles thereof are similar, and are not described herein again.
As shown in fig. 3, an embodiment of the present application provides another data aggregation device, where the device 30 includes: a listening module 301, an updating module 302, wherein,
the monitoring module 301 is configured to extract first target data according to a predetermined rule when a change in a data source of a target database is monitored;
here, the function of the listening module 301 in fig. 3 is the same as or similar to that of the listening module 201 in fig. 2.
And an updating module 302, configured to update data in the dimension table stored in the full-text search engine based on the extracted first target data.
Wherein the update module 302 of fig. 3 has the same or similar function as the update module 202 of fig. 2.
The embodiment of the present application provides a possible implementation manner, and specifically, the monitoring module 301 includes:
a determining unit 3011, configured to determine, based on a routing rule, a dimension model corresponding to each changed data, where the dimension model includes master table information, slave table information, master key information, and associated parameter information;
an obtaining unit 3012, configured to obtain, as first target data, associated parameters of corresponding master tables and/or slave tables from data that changes in a data source based on the master key information of all corresponding dimension models.
With the embodiment of the application, the problem of how to extract the first target data according to the predetermined rule is solved, that is, the problem of how to obtain the first target data when the data source in the database is monitored to be changed is solved.
The embodiment of the present application provides a possible implementation manner, and the updating module 302 is specifically configured to update, based on a type of data corresponding to the first target data, corresponding data in a dimension table on a full-text search engine, where the type includes master table data and slave table data.
According to the embodiment of the application, the problem of how to update the corresponding data in the dimension table on the full-text search engine according to the first target data is solved.
The embodiment of the present application provides a possible implementation manner, and further, the apparatus 30 further includes:
an extraction module 303, which extracts the second target data from the target database based on all the dimensional models;
and the storage module 304 is used for generating a dimension table based on the second target data and storing the dimension table to the full-text search engine.
According to the embodiment of the application, the problem of how to realize the integral aggregation of the historical data is solved.
The embodiment of the present application provides a possible implementation manner, and further, the apparatus 30 further includes:
a receiving module 305, configured to receive a report generation request and/or a BI analysis request;
and the analysis module 306 is used for generating reports and/or performing BI analysis based on the updated dimension table.
According to the embodiment of the application, the report can be generated in real time and/or the BI can be analyzed in real time.
The embodiments of the present application provide a possible implementation manner, where the target database includes a relational database and/or a non-relational database.
The embodiment of the application provides a possible implementation manner, wherein the full-text search engine is a Solr full-text search engine or an elastic search full-text search engine.
For the embodiment of the application, the dimension table obtained by aggregation is stored in a Solr full-text search engine or an elastic search full-text search engine, so that the problems of subsequent report generation and BI analysis are solved.
The embodiment of the application provides a data aggregation device, and compared with the prior art that time is consumed for obtaining latest data and efficiency is low, the data aggregation device extracts first target data according to a preset rule when a data source of a target database is monitored to change, and updates data in a dimension table stored in a full-text search engine based on the extracted first target data. That is, according to the embodiment of the present application, when it is monitored that a data source changes, the first target data is extracted according to a predetermined rule, and data in a dimension table stored in a full-text search engine is updated, in other words, only the changed data is aggregated, aggregation of the entire data is avoided, data processing amount is reduced, and real-time aggregation of the data can be achieved, so that efficiency of obtaining the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and information hysteresis of report generation or BI analysis is avoided.
The embodiments of the present application provide a data aggregation device, which is suitable for the method shown in the above embodiments, and details are not described herein again.
An embodiment of the present application provides an electronic device, as shown in fig. 4, an electronic device 40 shown in fig. 4 includes: a processor 4001 and a memory 4003. Processor 4001 is coupled to memory 4003, such as via bus 4002. Further, the electronic device 40 may also include a transceiver 4004. In addition, the transceiver 4004 is not limited to one in practical applications, and the structure of the electronic device 40 is not limited to the embodiment of the present application. The processor 4001 is applied in this embodiment of the application, and is configured to implement the functions of the monitoring module and the updating module shown in fig. 2 or fig. 3, and the functions of the extracting module, the storing module, the receiving module, and the analyzing module shown in fig. 3. The transceiver 4004 includes a receiver and a transmitter.
The memory 4003 is used for storing application codes for executing the scheme of the present application, and the execution is controlled by the processor 4001. The processor 4001 is configured to execute application code stored in the memory 4003 to implement the functions of the data aggregation apparatus provided by the embodiment shown in fig. 2 or fig. 3.
Compared with the prior art that time is consumed for obtaining the latest data and efficiency is low, the electronic equipment extracts the first target data according to the preset rule when the change of the data source of the target database is monitored, and updates the data in the dimension table stored by the full-text search engine based on the extracted first target data. That is, according to the embodiment of the present application, when it is monitored that a data source changes, the first target data is extracted according to a predetermined rule, and data in a dimension table stored in a full-text search engine is updated, in other words, only the changed data is aggregated, aggregation of the entire data is avoided, data processing amount is reduced, and real-time aggregation of the data can be achieved, so that efficiency of obtaining the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and information hysteresis of report generation or BI analysis is avoided.
The embodiment of the application provides an electronic device suitable for the method embodiment. And will not be described in detail herein.
The present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method shown in the above embodiments is implemented.
Compared with the prior art that the latest data acquisition is time-consuming and inefficient, the method and the device for updating the data in the dimension table stored by the full-text search engine extract the first target data according to the preset rule when the change of the data source of the target database is monitored, and update the data in the dimension table stored by the full-text search engine based on the extracted first target data. That is, according to the embodiment of the present application, when it is monitored that a data source changes, the first target data is extracted according to a predetermined rule, and data in a dimension table stored in a full-text search engine is updated, in other words, only the changed data is aggregated, aggregation of the entire data is avoided, data processing amount is reduced, and real-time aggregation of the data can be achieved, so that efficiency of obtaining the latest data is improved, further, report generation or BI analysis can be performed according to the latest data, and information hysteresis of report generation or BI analysis is avoided.
The embodiment of the application provides a computer-readable storage medium which is suitable for the method embodiment. And will not be described in detail herein.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.
Claims (10)
1. A data aggregation method, comprising:
when monitoring that the data source of the target database changes, extracting first target data according to a preset rule;
and updating the data in the dimension table stored by the full-text search engine based on the extracted first target data.
2. The method of claim 1, wherein the extracting the first target data according to a predetermined rule when the data source of the target database is monitored to be changed comprises:
determining a dimension model corresponding to each changed data based on a routing rule, wherein the dimension model comprises master table information, slave table information, master key information and associated parameter information;
and acquiring the associated parameters of corresponding master tables and/or slave tables from the changed data in the data source respectively based on the master key information of all the corresponding dimension models to serve as the first target data.
3. The method according to claim 1 or 2, wherein the updating data in the dimension table stored in the full text search engine based on the extracted first target data comprises:
and respectively updating corresponding data in the dimension table on the full-text search engine based on the type of the data corresponding to the first target data, wherein the type comprises master table data and slave table data.
4. A method according to claim 2, characterized in that the method comprises:
extracting second target data from the target database based on all dimension models;
and generating the dimension table based on the second target data and storing the dimension table to the full-text search engine.
5. The method according to any one of claims 1-4, characterized in that the method further comprises:
receiving a report generation request and/or a BI analysis request;
and generating a report and/or performing BI analysis based on the updated dimension table.
6. The method of claim 1, wherein the target database comprises a relational database and/or a non-relational database.
7. The method of any of claim 1, wherein the full-text search engine is a Solr full-text search engine or an elastic search full-text search engine.
8. A data convergence device, comprising:
the monitoring module is used for extracting first target data according to a preset rule when monitoring that a data source of the target database changes;
and the updating module is used for updating the data in the dimension table stored by the full-text search engine based on the extracted first target data.
9. An electronic device, comprising:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: performing the data aggregation method according to any one of claims 1 to 7.
10. A computer-readable storage medium for storing computer instructions which, when executed on a computer, cause the computer to perform the data aggregation method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010331096.5A CN111522918A (en) | 2020-04-24 | 2020-04-24 | Data aggregation method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010331096.5A CN111522918A (en) | 2020-04-24 | 2020-04-24 | Data aggregation method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111522918A true CN111522918A (en) | 2020-08-11 |
Family
ID=71912028
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010331096.5A Pending CN111522918A (en) | 2020-04-24 | 2020-04-24 | Data aggregation method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111522918A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113553488A (en) * | 2021-07-15 | 2021-10-26 | 挂号网（杭州）科技有限公司 | Method and device for updating index data in search engine, electronic equipment and medium |
| CN113990473A (en) * | 2021-10-28 | 2022-01-28 | 上海昆亚医疗器械股份有限公司 | Medical equipment operation and maintenance information collecting and analyzing system and using method thereof |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE60315291D1 (en) * | 2003-08-27 | 2007-09-13 | Sap Ag | Forward changes to a database |
| CN104424279A (en) * | 2013-08-30 | 2015-03-18 | 腾讯科技（深圳）有限公司 | Text relevance calculating method and device |
| CN108681593A (en) * | 2018-05-16 | 2018-10-19 | 青岛海信移动通信技术股份有限公司 | Service data retrieval method and device |
| CN110609844A (en) * | 2018-05-29 | 2019-12-24 | 优信拍（北京）信息科技有限公司 | Data updating method, device and system |
| CN110941641A (en) * | 2019-11-15 | 2020-03-31 | 南威软件股份有限公司 | Method for searching data across multiple databases |
| CN111026727A (en) * | 2019-10-24 | 2020-04-17 | 贝壳技术有限公司 | Table dimension retrieval data synchronization method, system and device based on log file |
-
2020
- 2020-04-24 CN CN202010331096.5A patent/CN111522918A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE60315291D1 (en) * | 2003-08-27 | 2007-09-13 | Sap Ag | Forward changes to a database |
| CN104424279A (en) * | 2013-08-30 | 2015-03-18 | 腾讯科技（深圳）有限公司 | Text relevance calculating method and device |
| CN108681593A (en) * | 2018-05-16 | 2018-10-19 | 青岛海信移动通信技术股份有限公司 | Service data retrieval method and device |
| CN110609844A (en) * | 2018-05-29 | 2019-12-24 | 优信拍（北京）信息科技有限公司 | Data updating method, device and system |
| CN111026727A (en) * | 2019-10-24 | 2020-04-17 | 贝壳技术有限公司 | Table dimension retrieval data synchronization method, system and device based on log file |
| CN110941641A (en) * | 2019-11-15 | 2020-03-31 | 南威软件股份有限公司 | Method for searching data across multiple databases |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113553488A (en) * | 2021-07-15 | 2021-10-26 | 挂号网（杭州）科技有限公司 | Method and device for updating index data in search engine, electronic equipment and medium |
| CN113990473A (en) * | 2021-10-28 | 2022-01-28 | 上海昆亚医疗器械股份有限公司 | Medical equipment operation and maintenance information collecting and analyzing system and using method thereof |
| CN113990473B (en) * | 2021-10-28 | 2022-09-30 | 上海昆亚医疗器械股份有限公司 | Medical equipment operation and maintenance information collecting and analyzing system and using method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6634527B2 (en) | User authority data search method and electronic device | |
| CN105122243B (en) | Expansible analysis platform for semi-structured data | |
| US8825581B2 (en) | Simplifying a graph of correlation rules while preserving semantic coverage | |
| US11755938B2 (en) | Graphical user interface indicating anomalous events | |
| CN103488673B (en) | For performing the method for reconciliation process, controller and data-storage system | |
| US8412696B2 (en) | Real time searching and reporting | |
| US8589432B2 (en) | Real time searching and reporting | |
| CN109313640B (en) | Method and system for database optimization | |
| US20180129708A1 (en) | Query processing management in a database management system | |
| US8880511B2 (en) | Database query optimization and cost estimation | |
| CN106156088B (en) | Index data processing method, data query method and device | |
| US11748351B2 (en) | Class specific query processing | |
| CN105389352A (en) | Log processing method and apparatus | |
| US20090327339A1 (en) | Partition templates for multidimensional databases | |
| CN111522918A (en) | Data aggregation method and device, electronic equipment and computer readable storage medium | |
| Theeten et al. | Chive: Bandwidth optimized continuous querying in distributed clouds | |
| Srivastava et al. | TBSAM: An access method for efficient processing of statistical queries | |
| CN113722600A (en) | Data query method, device, equipment and product applied to big data | |
| Xu et al. | Efficient similarity join based on Earth mover’s Distance using Mapreduce | |
| CN111797095A (en) | Index construction method and JSON data query method | |
| US20220215021A1 (en) | Data Query Method and Apparatus, Computing Device, and Storage Medium | |
| EP3436988B1 (en) | "methods and systems for database optimisation" | |
| CN112632118A (en) | Method, device, computing equipment and storage medium for querying data | |
| US11586604B2 (en) | In-memory data structure for data access | |
| US11841857B2 (en) | Query efficiency using merged columns |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200811 |
|
| RJ01 | Rejection of invention patent application after publication |