CN111026796A - Multi-source heterogeneous data acquisition method, device, system, medium and equipment - Google Patents
Multi-source heterogeneous data acquisition method, device, system, medium and equipment Download PDFInfo
- Publication number
- CN111026796A CN111026796A CN201911201662.4A CN201911201662A CN111026796A CN 111026796 A CN111026796 A CN 111026796A CN 201911201662 A CN201911201662 A CN 201911201662A CN 111026796 A CN111026796 A CN 111026796A
- Authority
- CN
- China
- Prior art keywords
- data
- acquisition
- data acquisition
- source
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses a multi-source heterogeneous data acquisition method, a device, a system, a medium and equipment, wherein the acquisition method comprises the following steps: firstly, establishing a keyword list; creating corresponding acquisition grammars according to the acquisition contents of the data sources, then respectively establishing data acquisition rules aiming at the acquisition grammars, and associating the data acquisition rules to the keywords corresponding to the keyword list; when data source collection is required, sending the data collection rule to a collection end, so that the collection end can collect data according to the corresponding data collection rule, and creating the data collection rule of a newly-appeared data source according to the currently-constructed data collection rule; in the method, the corresponding data acquisition rules are respectively constructed for each data source, so the method can realize acquisition for different data sources, solves the technical problems that the tool acquisition format is single and the acquisition needs to independently design an acquisition tool for a certain format in the prior art, and has universality, expansibility and reusability.
Description
Technical Field
The invention relates to the technical field of informatization, in particular to a multi-source heterogeneous data acquisition method, device, system, medium and equipment.
Background
In the enterprise information construction process, due to the influence of factors such as the stage, the technology, other economic factors and human factors of each business system construction and implementation of a data management system, a large amount of business data adopting different storage modes are accumulated in the enterprise in the development process, the adopted data management systems are quite different, and the business data form a heterogeneous data source of the enterprise from a simple file database to a complex network database. The multi-source data is an indispensable part for constructing an information system and is mainly divided into two types, one type is structured data which is usually stored in a database in the form of a data table, and the other type is unstructured data, such as factory equipment data and worker working condition data. Because the different data sources use different collectors, different communication protocols of the collectors are different, transmission rates are different, multiple senses exist particularly in communication, and the data formats collected in the data sources are different. Therefore, for various data sources, corresponding acquisition tools need to be adopted separately for acquisition, which brings great inconvenience to data acquisition, management and analysis.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings in the prior art, and provides a multi-source heterogeneous data acquisition method which can realize acquisition aiming at different data sources, solves the technical problems that the acquisition format of a tool is single and an acquisition tool needs to be designed for a certain format in the prior art, and has universality, expansibility and reusability.
The second purpose of the invention is to provide a multi-source heterogeneous data acquisition device.
A third object of the present invention is to provide a multi-source heterogeneous data acquisition system.
A fourth object of the present invention is to provide a storage medium.
It is a fifth object of the invention to provide a computing device.
The first purpose of the invention is to provide a multi-source heterogeneous data acquisition method, which comprises the following steps:
establishing a keyword table for storing keywords;
acquiring the acquired content of each data source, and creating an acquisition grammar according to the acquired content of each data source;
aiming at various current data sources, respectively establishing data acquisition rules according to acquisition grammars, and associating the data acquisition rules to corresponding keywords of a keyword table, wherein the corresponding keywords are keywords in the acquired content of the data sources;
aiming at a newly appeared data source, creating a data acquisition rule according to a currently constructed data acquisition rule;
when the data source is required to be acquired, the data acquisition rule corresponding to the data source is sent to the acquisition end of the data source, so that the acquisition end acquires data according to the corresponding data acquisition rule.
Preferably, the process of constructing the data collection rule for the newly emerging data source is as follows:
firstly, determining key words in the acquired content of the data source, then inquiring the key words in a key word table, acquiring data acquisition rules related to the key words aiming at the key words appearing in the key word table, and acquiring acquisition grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
Preferably, the data acquisition rule further includes configuration file content, where the configuration file content includes a name of a central server, an IP of the central server, a name of a data source generator, an IP of the data source generator, a name of a data cache server, and an IP address of the data cache server;
the method further comprises the steps of: and linking the database through the configuration file in the data acquisition rule, and sending the data to the linked database for storage when corresponding data are acquired through the data acquisition rule.
Preferably, the data collection rule further includes the following contents: collected data structures, collected derived data structures, collected saved data set types, database addresses, and rules for the database.
Preferably, the data acquisition rule is converted into a json data format and then transmitted to the acquisition end.
The second purpose of the invention is to provide a multi-source heterogeneous data acquisition method and a multi-source heterogeneous data acquisition device, wherein the device comprises
The keyword list building module is used for building a keyword list for storing keywords;
the acquisition module is used for acquiring the acquired content of each data source;
the acquisition grammar building module is used for creating acquisition grammars according to the acquired contents of the data sources;
the first data acquisition rule construction module is used for respectively establishing data acquisition rules according to the acquisition syntax;
the association module is used for associating the data acquisition rule to a keyword corresponding to the keyword table, wherein the corresponding keyword is a keyword in the acquired content of the data source;
the sending module is used for sending the data acquisition rule corresponding to the data source to the acquisition end of the data source when the data source is to be acquired, so that the acquisition end acquires data according to the corresponding data acquisition rule;
and the second data acquisition rule building module is used for building a data acquisition rule of a newly appeared data source according to the currently built data acquisition rule.
The third purpose of the invention is to provide a multisource heterogeneous data acquisition method, a multisource heterogeneous data acquisition system, the system includes central server and acquisition end server; the central server is connected with the acquisition end server through a network, and the acquisition end server is connected with each data acquisition device of the acquisition end;
the central server is used for realizing the multi-source heterogeneous data acquisition method of the first object of the invention;
the acquisition end server is used for realizing the multi-source heterogeneous data acquisition method of the first object of the invention and/or receiving a data acquisition rule sent by the central server; each data acquisition device is used for sending the data acquisition rule to the acquisition end; the data acquisition device is used for receiving the data acquired by each data acquisition device according to the data acquisition rule and sending the data to the central server.
Preferably, the system further comprises a distributed cache server, wherein the distributed cache server is connected with the acquisition end server and is used for caching the data received by the acquisition end server from the data acquisition equipment.
A fourth object of the present invention is to provide a storage medium, wherein the storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the multi-source heterogeneous data acquisition method according to the first object of the present invention.
The fifth objective of the present invention is to provide a computing device, which includes a processor and a memory for storing a processor-executable program, and when the processor executes the program stored in the memory, the multi-source heterogeneous data acquisition method according to the first objective of the present invention is implemented.
Compared with the prior art, the invention has the following advantages and effects:
(1) in the multi-source heterogeneous data acquisition method, firstly, a keyword list is established; creating corresponding acquisition grammars according to the acquisition contents of the data sources, then respectively establishing data acquisition rules aiming at the acquisition grammars, and associating the data acquisition rules to the keywords corresponding to the keyword list; when data source collection is required, the data collection rule is sent to the collection end, so that the collection end can collect data according to the corresponding data collection rule.
(2) According to the multi-source heterogeneous data acquisition method, aiming at a newly appeared data source, keywords in the acquisition content of the newly appeared data source are obtained, a data acquisition rule managed by the keywords is found out from a keyword list, and the acquisition grammar in the associated data acquisition rule is used for constructing the data acquisition rule of the newly appeared data source.
(3) In the multi-source heterogeneous data acquisition method, the data acquisition rule also comprises the content of the configuration file, and the object to be acquired and the storage address of the acquired data can be determined by the content of the configuration file, so that greater convenience is brought to the acquisition work of the multi-source heterogeneous data.
(4) According to the multi-source heterogeneous data acquisition method, the database is linked through the configuration file in the data acquisition rule, and when corresponding data are acquired through the data acquisition rule, the corresponding data are sent to the linked database for storage.
Drawings
FIG. 1 is a flow chart of a multi-source heterogeneous data acquisition method of the present invention.
FIG. 2 is a block diagram of the multi-source heterogeneous data acquisition device according to the present invention.
FIG. 3 is a block diagram of the architecture of the computing device of the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
Example 1
The embodiment discloses a multi-source heterogeneous data acquisition method, which is used for realizing heterogeneous multi-data-source acquisition, and as shown in fig. 1, the specific process is as follows:
s1, establishing a keyword table for storing keywords;
s2, acquiring the acquired content of each data source, and creating an acquisition grammar according to the acquired content of each data source; the collection syntax in this embodiment refers to some semantic actions performed on the collected content, including some data processing performed on the collected content.
Each data source refers to an object acquired by each acquisition device at an acquisition end, for example, the temperature acquired by a thermometer for a certain device, the data source is the device, and the acquired content of the data source is temperature data of the device.
S3, aiming at various current data sources, respectively establishing data acquisition rules according to acquisition grammars, and associating the data acquisition rules to keywords corresponding to the keyword table, wherein the corresponding keywords are keywords in the acquired content of the data sources; if the number of words in the data source collected content is not large, all the characters of the collected content can be used as a keyword.
In this embodiment, the data collection rule includes, in addition to the collected syntax, the content of the configuration file, the collected data structure, the collected derived data structure, the collected and stored data set type, the database address, and the rule of the database, where the content of the configuration file includes the name of the central server, the IP of the central server, the name of the data source generator IP, the name of the data cache server, and the IP address of the data cache server. In this embodiment, the database may be linked by a configuration file in the data acquisition rule, and when corresponding data is acquired by the data acquisition rule, the corresponding data is sent to the linked database for storage.
When data of one or two cameras or data of operation duration, speed, temperature, state, and the like of one motor are to be collected, in this embodiment, the data collection rule created according to the collection syntax may be the following code:
RULE _ NAME ═ MES warehouse Collection
VERSION＝1.0.0
Device _ Camera _1 ═ IP address
Device _ Camera _2 ═ IP address
Device_Motor＝COM1
Redis_IP＝IP
Redis_Name＝master
G->A+B+C+D
A->speed
B->running_Time
C->temperature
D->state
In the above-mentioned acquisition rule, G is the corresponding acquisition syntax, and A, B, C and D are the contents acquired by the camera.
S4, aiming at the newly appeared data source, creating a data acquisition rule according to the currently constructed data acquisition rule; the method specifically comprises the following steps:
for a newly appeared data source, firstly determining key words in the collected content of the data source, then inquiring the key words in a key word table, obtaining data collection rules related to the key words aiming at the key words appearing in the key word table, and obtaining collection grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
S5, when data sources are to be acquired, sending the data acquisition rules corresponding to the data sources to the acquisition ends of the data sources, so that the acquisition ends acquire data according to the corresponding data acquisition rules; in this embodiment, the data collection rule is converted into json data format before being transmitted to the collection terminal.
Example 2
The embodiment discloses a multi-source heterogeneous data acquisition device, as shown in FIG. 2, the device comprises
The keyword list building module is used for building a keyword list for storing keywords;
the acquisition module is used for acquiring the acquired content of each data source;
the acquisition grammar building module is used for creating acquisition grammars according to the acquired contents of the data sources;
the first data acquisition rule construction module is used for respectively establishing data acquisition rules according to the acquisition syntax;
the association module is used for associating the data acquisition rule to a keyword corresponding to the keyword table, wherein the corresponding keyword is a keyword in the acquired content of the data source;
the sending module is used for sending the data acquisition rule corresponding to the data source to the acquisition end of the data source when the data source is to be acquired, so that the acquisition end acquires data according to the corresponding data acquisition rule;
the second data acquisition rule building module is used for building a data acquisition rule for a newly appeared data source, and specifically comprises the following steps: firstly, determining key words in the acquired content of the data source, then inquiring the key words in a key word table, acquiring data acquisition rules related to the key words aiming at the key words appearing in the key word table, and acquiring acquisition grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
The multi-source heterogeneous data acquisition device in this embodiment corresponds to the multi-source heterogeneous data acquisition method in embodiment 1, and therefore specific implementation of each module can be referred to in embodiment 1 above, and is not described in detail herein; it should be noted that, the apparatus provided in this embodiment is only illustrated by dividing the functional modules, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
Example 3
The embodiment discloses a multi-source heterogeneous data acquisition system, which comprises a central server, an acquisition end server and a distributed cache server; the central server is connected with the acquisition end server through a network, the acquisition end server is connected with each data acquisition device of the acquisition end, and the distributed cache server is connected with the acquisition end server;
the central server is used in the multi-source heterogeneous data acquisition method described in embodiment 1, and includes:
establishing a keyword table for storing keywords;
acquiring the acquired content of each data source, and creating an acquisition grammar according to the acquired content of each data source;
aiming at various current data sources, respectively establishing data acquisition rules according to acquisition grammars, and associating the data acquisition rules to corresponding keywords of a keyword table, wherein the corresponding keywords are keywords in the acquired content of the data sources;
when data sources are required to be acquired, the data acquisition rules corresponding to the data sources are sent to the acquisition ends of the data sources, so that the acquisition ends acquire data according to the corresponding data acquisition rules;
aiming at a newly appeared data source, creating a data acquisition rule according to a currently constructed data acquisition rule; the method specifically comprises the following steps:
for a newly appeared data source, firstly determining key words in the collected content of the data source, then inquiring the key words in a key word table, obtaining data collection rules related to the key words aiming at the key words appearing in the key word table, and obtaining collection grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
The acquisition end server is used for realizing the multi-source heterogeneous data acquisition method in the embodiment 1 and/or receiving a data acquisition rule sent by the central server; each data acquisition device is used for sending the data acquisition rule to the acquisition end; the data acquisition device is used for receiving the data acquired by each data acquisition device according to the data acquisition rule and sending the data to the central server.
And the distributed cache server is used for caching the data received by the acquisition end server from the data acquisition equipment.
Example 4
The present embodiment discloses a storage medium, wherein the storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the multi-source heterogeneous data acquisition method of embodiment 1, as follows:
establishing a keyword table for storing keywords;
acquiring the acquired content of each data source, and creating an acquisition grammar according to the acquired content of each data source;
aiming at various current data sources, respectively establishing data acquisition rules according to acquisition grammars, and associating the data acquisition rules to corresponding keywords of a keyword table, wherein the corresponding keywords are keywords in the acquired content of the data sources;
when data sources are required to be acquired, the data acquisition rules corresponding to the data sources are sent to the acquisition ends of the data sources, so that the acquisition ends acquire data according to the corresponding data acquisition rules;
aiming at a newly appeared data source, creating a data acquisition rule according to a currently constructed data acquisition rule; the method specifically comprises the following steps:
for a newly appeared data source, firstly determining key words in the collected content of the data source, then inquiring the key words in a key word table, obtaining data collection rules related to the key words aiming at the key words appearing in the key word table, and obtaining collection grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
The storage medium in this embodiment may be a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), a usb disk, a removable hard disk, or other media.
Example 5
The present embodiment discloses a computing device, as shown in fig. 3, comprising a processor 1402, a memory, an input device 1403, a display 1404, and a network interface 1405, connected by a system bus 1401. The processor 1402 is configured to provide computing and control capabilities, the memory includes a nonvolatile storage medium 1406 and an internal memory 1407, the nonvolatile storage medium 1406 stores an operating system, a computer program, and a database, the internal memory 1407 provides an environment for running the operating system and the computer program in the nonvolatile storage medium 1406, and when the computer program is executed by the processor 1402, the multi-source heterogeneous data collection method described in embodiment 1 is implemented as follows:
establishing a keyword table for storing keywords;
acquiring the acquired content of each data source, and creating an acquisition grammar according to the acquired content of each data source;
aiming at various current data sources, respectively establishing data acquisition rules according to acquisition grammars, and associating the data acquisition rules to corresponding keywords of a keyword table, wherein the corresponding keywords are keywords in the acquired content of the data sources;
when data sources are required to be acquired, the data acquisition rules corresponding to the data sources are sent to the acquisition ends of the data sources, so that the acquisition ends acquire data according to the corresponding data acquisition rules;
aiming at a newly appeared data source, creating a data acquisition rule according to a currently constructed data acquisition rule; the method specifically comprises the following steps:
for a newly appeared data source, firstly determining key words in the collected content of the data source, then inquiring the key words in a key word table, obtaining data collection rules related to the key words aiming at the key words appearing in the key word table, and obtaining collection grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
The computing device in this embodiment may be a desktop computer, a notebook computer, a smart phone, a PDA handheld terminal, a tablet computer, or other terminal device with a processor function.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (10)
1. A multi-source heterogeneous data acquisition method is characterized by comprising the following steps:
establishing a keyword table for storing keywords;
acquiring the acquired content of each data source, and creating an acquisition grammar according to the acquired content of each data source;
aiming at various current data sources, respectively establishing data acquisition rules according to acquisition grammars, and associating the data acquisition rules to corresponding keywords of a keyword table, wherein the corresponding keywords are keywords in the acquired content of the data sources;
aiming at a newly appeared data source, creating a data acquisition rule according to a currently constructed data acquisition rule;
when the data source is required to be acquired, the data acquisition rule corresponding to the data source is sent to the acquisition end of the data source, so that the acquisition end acquires data according to the corresponding data acquisition rule.
2. The multi-source heterogeneous data acquisition method according to claim 1, wherein the process of constructing the data acquisition rule for the emerging data source is as follows:
firstly, determining key words in the acquired content of the data source, then inquiring the key words in a key word table, acquiring data acquisition rules related to the key words aiming at the key words appearing in the key word table, and acquiring acquisition grammars related to the key words; aiming at keywords which do not appear in the keyword list, newly creating a collected grammar according to the collected content corresponding to the keywords; and forming a data acquisition rule of a newly-appeared data source by the acquired acquisition grammar.
3. The multi-source heterogeneous data collection method according to claim 1, wherein the data collection rules further include configuration file contents, the configuration file contents include a center server name, a center server IP, a data source generator name IP, a data cache server name, and a data cache server IP address;
the method further comprises the steps of: and linking the database through the configuration file in the data acquisition rule, and sending the data to the linked database for storage when corresponding data are acquired through the data acquisition rule.
4. The multi-source heterogeneous data collection method according to claim 1, wherein the data collection rules further include the following: collected data structures, collected derived data structures, collected saved data set types, database addresses, and rules for the database.
5. The multi-source heterogeneous data acquisition method according to claim 1, wherein the data acquisition rule is converted into a json data format and then transmitted to an acquisition end.
6. A multi-source heterogeneous data acquisition device is characterized by comprising
The keyword list building module is used for building a keyword list for storing keywords;
the acquisition module is used for acquiring the acquired content of each data source;
the acquisition grammar building module is used for creating acquisition grammars according to the acquired contents of the data sources;
the first data acquisition rule construction module is used for respectively establishing data acquisition rules according to the acquisition syntax;
the association module is used for associating the data acquisition rule to a keyword corresponding to the keyword table, wherein the corresponding keyword is a keyword in the acquired content of the data source;
the sending module is used for sending the data acquisition rule corresponding to the data source to the acquisition end of the data source when the data source is to be acquired, so that the acquisition end acquires data according to the corresponding data acquisition rule;
and the second data acquisition rule building module is used for building a data acquisition rule of a newly appeared data source according to the currently built data acquisition rule.
7. A multi-source heterogeneous data acquisition system is characterized by comprising a central server and an acquisition end server; the central server is connected with the acquisition end server through a network, and the acquisition end server is connected with each data acquisition device of the acquisition end;
the central server is used for realizing the multi-source heterogeneous data acquisition method of any one of claims 1-5;
the acquisition end server is used for realizing the multi-source heterogeneous data acquisition method of any one of claims 1-5 and/or receiving a data acquisition rule sent by a central server; each data acquisition device is used for sending the data acquisition rule to the acquisition end; the data acquisition device is used for receiving the data acquired by each data acquisition device according to the data acquisition rule and sending the data to the central server.
8. The multi-source heterogeneous data acquisition system according to claim 7, further comprising a distributed cache server connected to the acquisition end server for caching data received by the acquisition end server from the data acquisition device.
9. A storage medium storing a computer program which, when executed by a processor, causes the processor to perform the multi-source heterogeneous data collection method of any one of claims 1-5.
10. A computing device comprising a processor and a memory for storing processor-executable programs, wherein the processor, when executing a program stored in the memory, implements the multi-source heterogeneous data collection method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911201662.4A CN111026796B (en) | 2019-11-29 | 2019-11-29 | Multi-source heterogeneous data acquisition method, device, system, medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911201662.4A CN111026796B (en) | 2019-11-29 | 2019-11-29 | Multi-source heterogeneous data acquisition method, device, system, medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111026796A true CN111026796A (en) | 2020-04-17 |
| CN111026796B CN111026796B (en) | 2023-05-16 |
Family
ID=70207261
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911201662.4A Active CN111026796B (en) | 2019-11-29 | 2019-11-29 | Multi-source heterogeneous data acquisition method, device, system, medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026796B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737529A (en) * | 2020-07-23 | 2020-10-02 | 北京东方通科技股份有限公司 | Multi-source heterogeneous data acquisition method |
| CN114661513A (en) * | 2022-04-18 | 2022-06-24 | 广州菩润信息科技有限公司 | Distributed multi-source data acquisition method, system, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092817A (en) * | 2013-01-18 | 2013-05-08 | 五八同城信息技术有限公司 | Data collection method and data collection device based on script engine |
| CN106528769A (en) * | 2016-11-04 | 2017-03-22 | 乐视控股（北京）有限公司 | Data acquisition method and apparatus |
| CN109460944A (en) * | 2018-12-14 | 2019-03-12 | 平安健康保险股份有限公司 | Core based on big data protects method, apparatus, equipment and readable storage medium storing program for executing |
| CN109992603A (en) * | 2019-04-04 | 2019-07-09 | 北京金堤科技有限公司 | A kind of data search method, device, electronic equipment and computer-readable medium |
| CN110414986A (en) * | 2019-06-21 | 2019-11-05 | 中国平安财产保险股份有限公司 | Cash register method for routing foundation and relevant device based on big data analysis |
-
2019
- 2019-11-29 CN CN201911201662.4A patent/CN111026796B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092817A (en) * | 2013-01-18 | 2013-05-08 | 五八同城信息技术有限公司 | Data collection method and data collection device based on script engine |
| CN106528769A (en) * | 2016-11-04 | 2017-03-22 | 乐视控股（北京）有限公司 | Data acquisition method and apparatus |
| CN109460944A (en) * | 2018-12-14 | 2019-03-12 | 平安健康保险股份有限公司 | Core based on big data protects method, apparatus, equipment and readable storage medium storing program for executing |
| CN109992603A (en) * | 2019-04-04 | 2019-07-09 | 北京金堤科技有限公司 | A kind of data search method, device, electronic equipment and computer-readable medium |
| CN110414986A (en) * | 2019-06-21 | 2019-11-05 | 中国平安财产保险股份有限公司 | Cash register method for routing foundation and relevant device based on big data analysis |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111737529A (en) * | 2020-07-23 | 2020-10-02 | 北京东方通科技股份有限公司 | Multi-source heterogeneous data acquisition method |
| CN114661513A (en) * | 2022-04-18 | 2022-06-24 | 广州菩润信息科技有限公司 | Distributed multi-source data acquisition method, system, equipment and storage medium |
| CN114661513B (en) * | 2022-04-18 | 2024-01-23 | 广州菩润信息科技有限公司 | Distributed multi-source data acquisition method, system, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111026796B (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110096655B (en) | Search result sorting method, device, equipment and storage medium | |
| CN109033403B (en) | Method, apparatus and storage medium for searching blockchain data | |
| CN109582289B (en) | Method, system, storage medium and processor for processing rule flow in rule engine | |
| CN109101607B (en) | Method, apparatus and storage medium for searching blockchain data | |
| US11934287B2 (en) | Method, electronic device and computer program product for processing data | |
| CN111026796B (en) | Multi-source heterogeneous data acquisition method, device, system, medium and equipment | |
| CN112130830B (en) | Interface generation method and device and electronic equipment | |
| US11068496B2 (en) | System and method for data management | |
| CN111385264A (en) | Communication service data access system and method | |
| CN111367870A (en) | Method, device and system for sharing picture book | |
| CN110990350B (en) | Log analysis method and device | |
| CN109086414B (en) | Method, apparatus and storage medium for searching blockchain data | |
| CN114995816A (en) | Business process configuration method and device, electronic equipment and readable storage medium | |
| CN103324567A (en) | App engine debugging method and debugging system | |
| CN111047434A (en) | Operation record generation method and device, computer equipment and storage medium | |
| CN108959294B (en) | Method and device for accessing search engine | |
| CN111488386B (en) | Data query method and device | |
| KR20210000041A (en) | Method and apparatus for analyzing log data in real time | |
| CN112328658A (en) | User profile data processing method, device, equipment and storage medium | |
| CN115757041B (en) | Method for collecting dynamically configurable multi-cluster logs and application | |
| CN113220760B (en) | Data processing method, device, server and storage medium | |
| CN111124923B (en) | Running state query method and device, server equipment and storage medium | |
| CN112600918B (en) | Industrial control edge big data efficient processing method and system based on BS architecture | |
| CN112199428B (en) | Application management system, method and terminal equipment of functional data block | |
| CN114153909B (en) | Database table structure translation method and device based on vector template engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |