CN110716938A - Data aggregation method and device, storage medium and electronic device - Google Patents

Data aggregation method and device, storage medium and electronic device Download PDF

Info

Publication number
CN110716938A
CN110716938A CN201910980137.0A CN201910980137A CN110716938A CN 110716938 A CN110716938 A CN 110716938A CN 201910980137 A CN201910980137 A CN 201910980137A CN 110716938 A CN110716938 A CN 110716938A
Authority
CN
China
Prior art keywords
task
data
data source
meta
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910980137.0A
Other languages
Chinese (zh)
Inventor
堵新政
张毅然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Mininglamp Software System Co ltd
Original Assignee
Beijing Mininglamp Software System Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Mininglamp Software System Co ltd filed Critical Beijing Mininglamp Software System Co ltd
Priority to CN201910980137.0A priority Critical patent/CN110716938A/en
Publication of CN110716938A publication Critical patent/CN110716938A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/25Fusion techniques

Abstract

The invention provides a data aggregation method and device, a storage medium and an electronic device, wherein the method comprises the following steps: obtaining meta-information of each data source, wherein the meta-information is used for indicating at least one of the following: the type of the data source, the position of the data source, and the data structure of the data source; and classifying and storing the data sources according to the meta information. The invention solves the problem that the data aggregation method in the related technology aims at a certain data source or a data source with a single structure, and enriches the application scenes of data aggregation.

Description

Data aggregation method and device, storage medium and electronic device
Technical Field
The invention relates to the field of computers, in particular to a data aggregation method and device, a storage medium and an electronic device.
Background
Data is growing exponentially today as information technology is developed. Each business system of the enterprise also generates data resources with different magnitudes in the operation process, and the data types are diversified, the structure is complicated, and the distribution is decentralized. The data aggregation is to organically aggregate multi-source, heterogeneous and dispersed data into a large data platform physically or logically for centralized storage and management, and meanwhile, data sharing among systems and effective circulation among applications can be realized. However, it becomes very important how to effectively converge the data with different sources, formats and structures. The data aggregation method in the prior art aims at a certain data source or a data source with a single structure, and cannot meet the multi-source and heterogeneous scenes of enterprises.
In view of the above problems in the related art, no effective solution exists at present.
Disclosure of Invention
The embodiment of the invention provides a data aggregation method and device, a storage medium and an electronic device, which aim to at least solve the problem that the data aggregation method in the related art aims at a certain data source or a data source with a single structure.
According to an embodiment of the present invention, there is provided a data aggregation method, including: obtaining meta-information of each data source, wherein the meta-information is used for indicating at least one of the following: the type of the data source, the position of the data source, and the data structure of the data source; and classifying and storing the data sources according to the meta information.
Optionally, after obtaining the meta information of each data source, the method further includes: creating a plurality of acquisition tasks, wherein each acquisition task corresponds to one data type; selecting a data source matched with the specified type from the data sources according to the meta-information, and executing the selected data source according to the acquisition task; and saving the execution result of the acquisition task.
Optionally, the task included in the collection task includes at least one of the following: creating a task, deleting a task, running a task, re-running and checking task details.
Optionally, the execution result includes at least one of: collecting starting time, ending time, triggering mode, collecting amount and target storage position.
Optionally, the method further comprises: and monitoring the condition of a plurality of acquisition tasks executing data sources in real time.
According to an embodiment of the present invention, there is provided a data aggregation apparatus including: an obtaining module, configured to obtain meta information of each data source, where the meta information is used to indicate at least one of: the type of the data source, the position of the data source, and the data structure of the data source; and the classification module is used for classifying and storing each data source according to the meta information.
Optionally, the apparatus further comprises: the system comprises a creating module, a searching module and a processing module, wherein the creating module is used for creating a plurality of acquisition tasks after acquiring the meta information of each data source, and each acquisition task corresponds to one data type; the processing module is used for selecting a data source matched with the specified type from the data sources according to the meta-information and executing the selected data source according to the acquisition task; and the storage module is used for storing the execution result of the acquisition task.
Optionally, the task included in the collection task includes at least one of the following: creating a task, deleting a task, running a task, re-running and checking task details.
Optionally, the execution result includes at least one of: collecting starting time, ending time, triggering mode, collecting amount and target storage position.
Optionally, the apparatus further comprises: and the monitoring module is used for monitoring the conditions of a plurality of acquisition task execution data sources in real time.
According to a further embodiment of the present invention, there is also provided a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
According to the invention, the meta-information of each data source is acquired, wherein the meta-information is used for indicating at least one of the following: the data source type, the data source position and the data structure of the data source are classified and stored according to the meta information, so that the problem that the data aggregation method in the related technology aims at a certain data source or a data source with a single structure is solved, and the application scene of data aggregation is enriched.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a block diagram of a hardware structure of a terminal of a data aggregation method according to an embodiment of the present invention;
FIG. 2 is a flow diagram of a method of data aggregation according to an embodiment of the invention;
fig. 3 is a block diagram of a data aggregation apparatus according to an embodiment of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
Example 1
The method provided by the first embodiment of the present application may be executed in a terminal, a computer terminal, or a similar computing device. Taking an example of the operation on a terminal, fig. 1 is a hardware structure block diagram of the terminal of the data aggregation method according to the embodiment of the present invention. As shown in fig. 1, the terminal 10 may include one or more (only one shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, and optionally may also include a transmission device 106 for communication functions and an input-output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the terminal. For example, the terminal 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data aggregation method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the terminal 10. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In this embodiment, a method for aggregating data running on the above terminal or is provided, and fig. 2 is a flowchart of the method for aggregating data according to the embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, obtaining meta-information of each data source, wherein the meta-information is used for indicating at least one of the following: the type of the data source, the position of the data source, and the data structure of the data source;
and step S204, classifying and storing each data source according to the meta information.
Through the above steps S202 to S204, obtaining meta information of each data source is adopted, where the meta information is used to indicate at least one of the following: the data source type, the data source position and the data structure of the data source are classified and stored according to the meta information, so that the problem that the data aggregation method in the related technology aims at a certain data source or a data source with a single structure is solved, and the application scene of data aggregation is enriched.
It should be noted that the execution subject of the method steps of the present embodiment may be a big data platform.
In an optional implementation manner of this embodiment, after obtaining the meta information of each data source, the method of this embodiment may further include:
step S206, a plurality of acquisition tasks are created, wherein each acquisition task corresponds to one data type;
step S208, selecting a data source matched with the specified type from all data sources according to the meta-information, and executing the selected data source according to the acquisition task;
step S210, storing the execution result of the acquisition task;
step S212, the situation of a plurality of acquisition task execution data sources is monitored in real time.
Wherein, the task that the collection task includes one of the following at least: creating a task, deleting a task, running a task, re-running and checking task details.
Wherein, the execution result at least comprises one of the following: collecting starting time, ending time, triggering mode, collecting amount and target storage position.
The present application will be illustrated with reference to alternative embodiments of the present embodiment;
this optional embodiment provides a general data aggregation method, where the storage of data aggregation depends on an existing big data platform, and the method includes the following steps:
step S11, obtaining the meta-information of each data source needing to be converged to the big data platform, wherein the meta-information comprises data source type, data position, data structure and the like, and storing the information in a database;
step S12, determining a data acquisition program, namely a data source collector, aiming at the common data source,
wherein the collector includes: a common relational database (mysql, oracle, sqlserver, hive, postgresql and the like) collector, a semi-structured data (text, json file, excel and the like) collector and a streaming data collector;
step S13, a task manager is realized for managing the collection tasks, including task creation, task deletion, task operation, task re-operation, task detail checking and the like;
step S14, a task monitor is realized for monitoring the execution condition of the task;
step S16, implementing a task recorder for recording the execution result of the collection task including the collection start time, end time, trigger mode (automatic, manual), collection amount, target storage position and other information
Step S17, storage management after data aggregation, classified storage of data from different sources and maintenance of data versions;
the following description is made by taking specific embodiments as examples
Assuming that a Mysql data source Mysql-s exists, a big data platform BigDataPlatform is a distributed Hadoop platform, and firstly, data in the Mysql-s are gathered into the big data platform BigDataPlatform; acquiring all library information under Mysql-s, all list information under the library and all column information in the list; the Task manager creates a Mysql-type acquisition Task-1 and waits for the Task to be executed; according to the triggering mode of the task, the acquisition task is manually or automatically operated; calling a Mysql collector to extract data in Mysql-s to a big data platform according to the fact that the data source type of the collection task is the Mysql type; the Task monitor is started, the execution condition of the Task-1 is monitored in real time, and the execution condition is fed back to a user; after the Task-1 is executed, the Task recorder stores the Task execution result into the database; and performing classified storage according to a service system to which Mysql-s belongs, and simultaneously performing version maintenance on the data storage directory according to the data acquisition date.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
Example 2
In this embodiment, a data aggregation device is further provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and details of which have been already described are omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 3 is a block diagram of a data aggregation apparatus according to an embodiment of the present invention, and as shown in fig. 3, the apparatus includes: an obtaining module 32, configured to obtain meta information of each data source, where the meta information is used to indicate at least one of: the type of the data source, the position of the data source, and the data structure of the data source; and the classification module 34 is coupled to the obtaining module 32 and configured to perform classification storage on each data source according to the meta information.
Optionally, the apparatus in this embodiment may further include: the system comprises a creating module, a searching module and a processing module, wherein the creating module is used for creating a plurality of acquisition tasks after acquiring the meta information of each data source, and each acquisition task corresponds to one data type; the processing module is used for selecting a data source matched with the specified type from all data sources according to the meta-information and executing the selected data source according to the acquisition task; and the storage module is used for storing the execution result of the acquisition task.
Wherein, the task that the collection task includes one of the following at least: creating a task, deleting a task, running a task, re-running and checking task details.
Wherein, the execution result at least comprises one of the following: collecting starting time, ending time, triggering mode, collecting amount and target storage position.
Optionally, the apparatus of this embodiment may further include: and the monitoring module is used for monitoring the condition of a plurality of acquisition task execution data sources in real time.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Example 3
Embodiments of the present invention also provide a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the storage medium may be configured to store a computer program for executing the steps of:
s1, obtaining meta-information of each data source, wherein the meta-information is used for indicating at least one of the following: the type of the data source, the position of the data source, and the data structure of the data source;
and S2, classifying and storing the data sources according to the meta information.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, obtaining meta-information of each data source, wherein the meta-information is used for indicating at least one of the following: the type of the data source, the position of the data source, and the data structure of the data source;
and S2, classifying and storing the data sources according to the meta information.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.

Claims (12)

1. A method for aggregating data, comprising:
obtaining meta-information of each data source, wherein the meta-information is used for indicating at least one of the following: the type of the data source, the position of the data source, and the data structure of the data source;
and classifying and storing the data sources according to the meta information.
2. The method of claim 1, wherein after obtaining the meta information of each data source, the method further comprises:
creating a plurality of acquisition tasks, wherein each acquisition task corresponds to one data type;
selecting a data source matched with the specified type from the data sources according to the meta-information, and executing the selected data source according to the acquisition task;
and saving the execution result of the acquisition task.
3. The method of claim 2, wherein the tasks included in the collection task include at least one of: creating a task, deleting a task, running a task, re-running and checking task details.
4. The method of claim 2, wherein the execution result comprises at least one of: collecting starting time, ending time, triggering mode, collecting amount and target storage position.
5. The method of claim 2, further comprising:
and monitoring the condition of a plurality of acquisition tasks executing data sources in real time.
6. An apparatus for aggregating data, comprising:
an obtaining module, configured to obtain meta information of each data source, where the meta information is used to indicate at least one of: the type of the data source, the position of the data source, and the data structure of the data source;
and the classification module is used for classifying and storing each data source according to the meta information.
7. The apparatus of claim 6, further comprising:
the system comprises a creating module, a searching module and a processing module, wherein the creating module is used for creating a plurality of acquisition tasks after acquiring the meta information of each data source, and each acquisition task corresponds to one data type;
the processing module is used for selecting a data source matched with the specified type from the data sources according to the meta-information and executing the selected data source according to the acquisition task;
and the storage module is used for storing the execution result of the acquisition task.
8. The apparatus of claim 7, wherein the collection task comprises a task that includes at least one of: creating a task, deleting a task, running a task, re-running and checking task details.
9. The apparatus of claim 7, wherein the execution result comprises at least one of: collecting starting time, ending time, triggering mode, collecting amount and target storage position.
10. The apparatus of claim 7, further comprising:
and the monitoring module is used for monitoring the conditions of a plurality of acquisition task execution data sources in real time.
11. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to carry out the method of any one of claims 1 to 5 when executed.
12. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 5.
CN201910980137.0A 2019-10-15 2019-10-15 Data aggregation method and device, storage medium and electronic device Pending CN110716938A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910980137.0A CN110716938A (en) 2019-10-15 2019-10-15 Data aggregation method and device, storage medium and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910980137.0A CN110716938A (en) 2019-10-15 2019-10-15 Data aggregation method and device, storage medium and electronic device

Publications (1)

Publication Number Publication Date
CN110716938A true CN110716938A (en) 2020-01-21

Family

ID=69212631

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910980137.0A Pending CN110716938A (en) 2019-10-15 2019-10-15 Data aggregation method and device, storage medium and electronic device

Country Status (1)

Country Link
CN (1) CN110716938A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782906A (en) * 2020-06-29 2020-10-16 杭州海康威视数字技术股份有限公司 Data object aggregation method and device, electronic equipment and storage medium
CN114070787A (en) * 2021-11-15 2022-02-18 南京航空航天大学 Police service big data oriented data aggregation method and device, storage medium and electronic equipment

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104111983A (en) * 2014-06-30 2014-10-22 中国科学院信息工程研究所 Open-type multi-source data collection system and method
CN108182233A (en) * 2017-12-27 2018-06-19 苏州麦迪斯顿医疗科技股份有限公司 A kind of distributed data abstracting method, device, computer equipment and storage medium
CN109522312A (en) * 2018-11-27 2019-03-26 北京锐安科技有限公司 A kind of data processing method, device, server and storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104111983A (en) * 2014-06-30 2014-10-22 中国科学院信息工程研究所 Open-type multi-source data collection system and method
CN108182233A (en) * 2017-12-27 2018-06-19 苏州麦迪斯顿医疗科技股份有限公司 A kind of distributed data abstracting method, device, computer equipment and storage medium
CN109522312A (en) * 2018-11-27 2019-03-26 北京锐安科技有限公司 A kind of data processing method, device, server and storage medium

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111782906A (en) * 2020-06-29 2020-10-16 杭州海康威视数字技术股份有限公司 Data object aggregation method and device, electronic equipment and storage medium
CN114070787A (en) * 2021-11-15 2022-02-18 南京航空航天大学 Police service big data oriented data aggregation method and device, storage medium and electronic equipment
CN114070787B (en) * 2021-11-15 2022-10-04 南京航空航天大学 Police service big data oriented data aggregation method and device, storage medium and electronic equipment

Similar Documents

Publication Publication Date Title
CN108009236B (en) Big data query method, system, computer and storage medium
US9953066B2 (en) Automatically generating and reporting merged performance and capacity statistics
CN110427368A (en) Data processing method, device, electronic equipment and storage medium
US10133779B2 (en) Query hint management for a database management system
CN112506870B (en) Data warehouse increment updating method and device and computer equipment
US10929370B2 (en) Index maintenance management of a relational database management system
CN110912757B (en) Service monitoring method and server
CN114691786A (en) Method and device for determining data blood relationship, storage medium and electronic device
CN110858192A (en) Log query method and system, log checking system and query terminal
CN110716938A (en) Data aggregation method and device, storage medium and electronic device
CN110309206B (en) Order information acquisition method and system
CN107422991B (en) Storage strategy management system
CN110688355A (en) Method and device for changing container state
CN113704203A (en) Log file processing method and device
CN114490865A (en) Database synchronization method, device, equipment and computer storage medium
CN113779017A (en) Method and apparatus for data asset management
CN112000456A (en) Task scheduling method, device, equipment and storage medium
CN114040021B (en) Development processing method and device for application program
KR20210050827A (en) An extraction-system using dispersion deep learning information analysis management based cloud and method of it
CN110362706B (en) Data searching method and device, storage medium and electronic device
CN117349323B (en) Database data processing method and device, storage medium and electronic equipment
CN113434402A (en) Performance analysis method, device and equipment for micro-service application
CN116010340A (en) Data table management method and device
CN113434492A (en) Data detection method and device, storage medium and electronic device
CN111930718A (en) Node adjustment method and device for configuration management database

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200121