CN107330089B - Cross-network structured data collection system - Google Patents
Cross-network structured data collection system Download PDFInfo
- Publication number
- CN107330089B CN107330089B CN201710538423.2A CN201710538423A CN107330089B CN 107330089 B CN107330089 B CN 107330089B CN 201710538423 A CN201710538423 A CN 201710538423A CN 107330089 B CN107330089 B CN 107330089B
- Authority
- CN
- China
- Prior art keywords
- data
- communication
- data center
- communication request
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013480 data collection Methods 0.000 title claims abstract description 28
- 238000004891 communication Methods 0.000 claims abstract description 186
- 230000005540 biological transmission Effects 0.000 claims abstract description 10
- 239000008358 core component Substances 0.000 claims description 26
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000000034 method Methods 0.000 abstract description 14
- 230000008569 process Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013075 data extraction Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 239000000306 component Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
The embodiment of the invention provides a cross-network structured data collection system. The system comprises: the data loading service module and the data center communication module; the data loading service module is used for collecting data in each database of the source data center and generating a first communication request; the data center communication module is used for sending the first communication request to a target data center; receiving feedback information of a target data center; and sending the data to a destination data center. The embodiment of the invention forms a peer-to-peer network subset by establishing the cross-network structured data collection system at each molecular company. The pressure of a headquarter data center in the data transmission and exchange process is relieved, and the efficiency of the molecular company is improved. A set of data collection system for collecting data and storing in a distributed manner in each data center is formed, and a communication mode of the molecular data center is established, so that network bandwidth resources are saved, and a basis is provided for data exchange and use of the molecular data center.
Description
Technical Field
The embodiment of the invention relates to the technical field of communication, in particular to a cross-network structured data collection system.
Background
With the continuous mining of the potential of big data applications, many molecular companies with companies distributed in different regions have their own data centers. The data department needs to collect and integrate the data of the data centers across the public network to provide data services for the whole company.
In the prior art, a molecular company collects data respectively and gathers the data to a headquarter data center, and the headquarter data center performs unified fusion on the data to form data assets.
Because a certain time is consumed when the headquarter data center performs unified fusion on the data, the molecular company can access and acquire the fused data only after the headquarter data center performs unified fusion on the data, and the efficiency of the molecular company is reduced. In addition, when different molecule companies need to send data to the headquarters data center, one molecule company needs to wait for the other molecule company to send the data after sending the data, that is, different molecule companies cannot perform data sending well concurrently, which results in waste of network bandwidth resources.
Disclosure of Invention
The embodiment of the invention provides a cross-network structured data collection system, which is used for improving the efficiency of molecular companies and network bandwidth resources.
One aspect of an embodiment of the present invention is to provide a cross-network structured data collection system, including: the data loading service module and the data center communication module;
the data loading service module is used for collecting data in each database of a source data center and generating a first communication request, and the first communication request is used for requesting the data center communication module to send the data to a target data center;
the data center communication module is used for sending the first communication request to the target data center; receiving feedback information of the target data center; and sending the data to the destination data center.
Optionally, the data loading service module is further configured to record a working state of the data loading service module, where the working state includes at least one of the following:
data acquisition state, data preprocessing state and data transmission state.
Optionally, the data center communication module includes: a communication output unit and a communication core component;
the communication output unit is used for transmitting the first communication request to the communication core component;
the communication core component is used for converting the format of the first communication request and sending the converted first communication request to the target data center.
Optionally, the data center communication module further includes: a communication input unit;
the communication input unit is used for receiving a second communication request sent by the target data center and verifying the access authority of the target data center according to the second communication request.
Optionally, when the communication input unit verifies the access right of the destination data center according to the second communication request, the communication input unit is specifically configured to:
analyzing the second communication request to obtain a uniform resource locator in the second communication request;
inquiring whether the data center communication module stores the uniform resource locator;
and if the data center communication module stores the uniform resource locator, successfully verifying the access authority of the target data center.
Optionally, the data center communication module stores a white list, where the white list includes an access rule of an application programming interface;
when the communication input unit queries whether the data center communication module stores the uniform resource locator, the communication input unit is specifically configured to:
and inquiring whether the access rule of the application programming interface corresponding to the uniform resource locator is stored in the white list.
Optionally, after the communication input unit successfully verifies the access right of the destination data center, the communication input unit is further configured to:
transmitting the second communication request to the communication core component.
Optionally, the data center communication module further includes: an application program;
the communication core component is further configured to transmit the second communication request to the application.
Optionally, the application program is further configured to send the first communication request to the communication output unit.
Optionally, the data loading service module includes: a data loading engine;
the data loading engine is used for periodically collecting data in each database of the source data center.
The cross-network structured data collection system provided by the embodiment of the invention forms a peer-to-peer network subset by establishing the cross-network structured data collection system in each molecular company. The pressure of a headquarter data center in the data transmission and exchange process is relieved, and the efficiency of the molecular company is improved. A set of data collection system for collecting data and storing in a distributed manner in each data center is formed, and a communication mode of the molecular data center is established, so that network bandwidth resources are saved, and a basis is provided for data exchange and use of the molecular data center.
Drawings
FIG. 1 is a diagram illustrating a relational database collection method in the prior art;
FIG. 2 is a diagram illustrating a prior art method for collecting data and storing the data in a distributed manner by open source software;
FIG. 3 is a diagram illustrating the mutual conductance of a relational database and distributed storage data in the prior art;
FIG. 4 is a prior art schematic of molecular company data collection;
FIG. 5 is a schematic diagram of a cross-network structured data collection system provided by an embodiment of the present invention;
fig. 6 is a schematic view of a work flow of the data loading service module provided in this embodiment;
fig. 7 is a schematic view of a work flow of the data center communication module provided in this embodiment.
Detailed Description
With the continuous mining of the potential of big data applications, many molecular companies with companies distributed in different regions have their own data centers. The data department needs to collect and integrate the data of the data centers across the public network to provide data services for the whole company.
In the prior art, each subdata center has its own independent database, and if the massive data needs to be effectively analyzed, the data from each molecular company database needs to be imported into a distributed storage cluster or a centralized large-scale distributed database. This process involves the collection of data and the concentration of data, two phases. There are several ways for molecular companies to collect data:
one way is that: as shown in fig. 1, a conventional relational database (Oracle, MySQL) is built. Corresponding data storage is established through commands of a traditional relational database product or some middleware (PL/SQL Developer and the like), and corresponding data is imported and stored.
The other mode is as follows: many enterprises have their own data collection tools, and are mostly used for collecting system logs. Such as the flare of Apache Chukwa, Cloudera, etc., these data collection tools all adopt a distributed architecture, and can meet the data acquisition and data transmission requirements of hundreds of MB per second. The data collected by these collection tools is typically stored in a hadoop distributed file system, HDFS, as shown in fig. 2.
Yet another way is: as shown in fig. 3, the enterprise may also choose to utilize sqoop to transfer data from the conventional database to the HDFS, and perform mutual migration between hadoop and data in the conventional relational database. The Sqoop provides interfaces between various HDFS and a general data system, executes in a command line parameter mode, and writes data in a relational database (MySQL, Oracle) into a hadoop distributed file system. And data in the hadoop distributed file system can also be imported into the relational database.
As shown in fig. 4, each molecular company collects data in the above manner to form a conventional relational database or HDFS storage, i.e., its own data center. And thus large data sets that can be migrated to the headquarters data center. And the data can be fused to form data assets after being collected to a headquarter data center, so that higher-quality data service is provided.
Fig. 5 is a schematic diagram of a cross-network structured data collection system according to an embodiment of the present invention. As shown in fig. 5, the cross-network structured data collection system provided by the present embodiment mainly includes two large modules: the data loading service module and the data center communication module; the data loading service module is used for collecting data in each database of a source data center and generating a first communication request, and the first communication request is used for requesting the data center communication module to send the data to a target data center; the data center communication module is used for sending the first communication request to the target data center; receiving feedback information of the target data center; and sending the data to the destination data center.
The data loading service module is the core of a cross-network structured data collection system, and has the primary functions of providing a timed data collection task and further providing the functions of manually triggering tasks, rerunning all failed tasks, recollecting specific data and the like according to actual application. The following is specifically described:
1. and (3) manually triggering a task: the user may manually initiate all data collection tasks. If a fault is encountered, which results in service not being able to be started, a mechanism for manually starting the task is required.
2. Rerun all failed tasks: the data loading service module is further configured to record a working state of the data loading service module, where the working state includes at least one of: data acquisition state, data preprocessing state and data transmission state.
Each stage of the data loading service module has a corresponding record, and the stage of the task is recorded, namely the data extraction stage, the data preprocessing stage or the data transmission stage. The data loading service module sets a monitor for each task thread, and the monitor is responsible for writing the state of the task into a database in time.
3. The specific data is collected again: providing for manipulation of specific data in the database for a property avoids restarting all data loading tasks for certain individual data.
4. Starting a collection task: the primary role of the data load service module is to automatically start the task of performing data collection at a certain time. It is a timed task.
5. Self-checking: the self state needs to be reported regularly: whether it is operating normally.
The main role of the data center communication module is to provide a solution for communication between data centers, and the main task of the data center communication module is to provide transparent communication service for upper-layer application programs (referred to as a data loading service module here, but it can be conveniently extended to support other applications). The application program does not need to know whether the target data application is in the data center, and the data in other data centers are accessed in the same way as the data center without change.
In this embodiment, the data center communication module includes: a communication output unit and a communication core component; the communication output unit is used for transmitting the first communication request to the communication core component; the communication core component is used for converting the format of the first communication request and sending the converted first communication request to the target data center. The data center communication module further comprises: a communication input unit; the communication input unit is used for receiving a second communication request sent by the target data center and verifying the access authority of the target data center according to the second communication request. When the communication input unit verifies the access right of the destination data center according to the second communication request, the communication input unit is specifically configured to: analyzing the second communication request to obtain a uniform resource locator in the second communication request; inquiring whether the data center communication module stores the uniform resource locator; and if the data center communication module stores the uniform resource locator, successfully verifying the access authority of the target data center. The data center communication module stores a white list, and the white list comprises access rules of an application programming interface; when the communication input unit queries whether the data center communication module stores the uniform resource locator, the communication input unit is specifically configured to: and inquiring whether the access rule of the application programming interface corresponding to the uniform resource locator is stored in the white list. After the communication input unit successfully verifies the access right of the destination data center, the communication input unit is further configured to: transmitting the second communication request to the communication core component. The data center communication module further comprises: an application program; the communication core component is further configured to transmit the second communication request to the application. The application program is further configured to send the first communication request to the communication output unit.
The functions of the communication output unit, the communication core component and the communication input unit are respectively described in detail as follows:
communication output unit (out): the data loading service module extracts and converts data and then calls the service, and the communication output unit (out) is responsible for transferring the request to a communication core component (Telcore) for further processing. The communication core component (Telcore) receives the call of the communication output unit (out), redirects the communication request of the user, and maps the request Uniform Resource Locator (URL) in the data center into the URL which can be transmitted on the network. And assembling a header field of the new request message and a main body of the original message into a new HyperText Transfer Protocol (HTTP) request. The request is then sent.
Communication input unit (in): the data center communication module is mainly responsible for receiving communication request processing and access control initiated by a communication core component (Telcore), a data center communication module monitors requests from other data centers after being started, when the requests from other data centers are captured, a communication input unit (in) firstly analyzes a request message, extracts a header field of the message to check a URL (uniform resource locator) in the message, compares the configuration of the data center communication module, and judges whether the record exists in a white list (the white list is a configuration file, and a series of API access rules are stipulated in the white list). If the white list exists, the message is converted into a message in the data center and delivered to an upper application program. And the target application program receives the request, processes the request and sends the result to a communication module in the data center. The communication output unit (out) of the communication module receives the request of the application program and then performs corresponding operation. So far, the communication is successful. If the white list does not exist, the communication fails.
The embodiment of the invention forms a peer-to-peer network subset by establishing the cross-network structured data collection system at each molecular company. The pressure of a headquarter data center in the data transmission and exchange process is relieved, and the efficiency of the molecular company is improved. A set of data collection system for collecting data and storing in a distributed manner in each data center is formed, and a communication mode of the molecular data center is established, so that network bandwidth resources are saved, and a basis is provided for data exchange and use of the molecular data center.
Fig. 6 is a schematic view of a work flow of the data loading service module provided in this embodiment. As shown in fig. 6, the data loading service module includes: a data loading engine; the data loading engine is used for periodically collecting data in each database of the source data center.
The data loading engine is responsible for the task scheduling of the whole data extraction. Timing trigger jobs (jobs) extract data, convert data formats, and send data. The information of the tasks is configured through an XML file and mainly comprises information such as storage data, data centers which belong to the tasks, data sources, and the data volume which can be extracted by each task (task) started by the job.
Fig. 7 is a schematic view of a work flow of the data center communication module provided in this embodiment. As shown in fig. 7, data center a is a source data center and data center B is a destination data center. The data center a includes: the system comprises a communication input unit, a communication output unit, a communication core component and a source application program; the data center B includes: the system comprises a communication input unit, a communication output unit, a communication core component and a target application program. When a source application of data center a communicates with a destination application of data center B, the source application of data center a first sends a communication request to the communication output unit, where the source application may be a program that has a demand for data, including a demand for transmitting data and using data. The destination application may be data only or may be the result of running within the application. After receiving the communication request, the communication output unit of the data center a forwards the communication request to the communication core component of the data center a, and the communication core component of the data center a sends the communication request to the data center B. After capturing a communication request sent by the data center A, the communication input unit of the data center B detects the access authority of the data center A, and if the data center A has the authority to access the data center B, the communication input unit of the data center B forwards the communication request to a destination application program through the communication core component of the data center B. After receiving the communication request, the destination application program processes the communication request and sends a corresponding reply to the communication output unit of the data center B, the communication output unit of the data center B sends the corresponding reply to the communication input unit of the data center A through the communication core component of the data center B, and the communication input unit of the data center A sends the corresponding reply to the source application program through the communication core component of the data center A, so that the communication between the source application program and the destination application program is realized.
The embodiment of the invention forms a peer-to-peer network subset by establishing the cross-network structured data collection system at each molecular company. The pressure of a headquarter data center in the data transmission and exchange process is relieved, and the efficiency of the molecular company is improved. A set of data collection system for collecting data and storing in a distributed manner in each data center is formed, and a communication mode of the molecular data center is established, so that network bandwidth resources are saved, and a basis is provided for data exchange and use of the molecular data center.
In summary, the embodiments of the present invention form a peer-to-peer network subset by building the above-mentioned cross-network structured data collection system at each molecular company. The pressure of a headquarter data center in the data transmission and exchange process is relieved, and the efficiency of the molecular company is improved. A set of data collection system for collecting data and storing in a distributed manner in each data center is formed, and a communication mode of the molecular data center is established, so that network bandwidth resources are saved, and a basis is provided for data exchange and use of the molecular data center.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It is obvious to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to perform all or part of the above described functions. For the specific working process of the device described above, reference may be made to the corresponding process in the foregoing method embodiment, which is not described herein again.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (9)
1. A cross-network structured data collection system, comprising: the data loading service module and the data center communication module;
the data loading service module is used for collecting data in each database of a source data center and generating a first communication request, and the first communication request is used for requesting the data center communication module to send the data to a target data center;
the data center communication module is used for sending the first communication request to the target data center; receiving feedback information of the target data center; sending the data to the target data center;
the data center communication module comprises: a communication output unit and a communication core component;
the communication output unit is used for transmitting the first communication request to the communication core component;
the communication core component is used for converting the format of the first communication request and sending the converted first communication request to the target data center.
2. The system of claim 1, wherein the data loading service module is further configured to record an operating status thereof, and the operating status includes at least one of:
data acquisition state, data preprocessing state and data transmission state.
3. The system of claim 1 or 2, wherein the data center communication module further comprises: a communication input unit;
the communication input unit is used for receiving a second communication request sent by the target data center and verifying the access authority of the target data center according to the second communication request.
4. The system according to claim 3, wherein when the communication input unit verifies the access right of the destination data center according to the second communication request, the communication input unit is specifically configured to:
analyzing the second communication request to obtain a uniform resource locator in the second communication request;
inquiring whether the data center communication module stores the uniform resource locator;
and if the data center communication module stores the uniform resource locator, successfully verifying the access authority of the target data center.
5. The system of claim 4, wherein the data center communication module stores a white list, the white list including access rules for application programming interfaces;
when the communication input unit queries whether the data center communication module stores the uniform resource locator, the communication input unit is specifically configured to:
and inquiring whether the access rule of the application programming interface corresponding to the uniform resource locator is stored in the white list.
6. The system according to claim 4 or 5, wherein after the communication input unit successfully verifies the access right of the destination data center, the system is further configured to:
transmitting the second communication request to the communication core component.
7. The system of claim 6, wherein the data center communication module further comprises: an application program;
the communication core component is further configured to transmit the second communication request to the application.
8. The system of claim 7, wherein the application is further configured to send the first communication request to the communication output unit.
9. The system of claim 1, wherein the data loading service module comprises: a data loading engine;
the data loading engine is used for periodically collecting data in each database of the source data center.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710538423.2A CN107330089B (en) | 2017-07-04 | 2017-07-04 | Cross-network structured data collection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710538423.2A CN107330089B (en) | 2017-07-04 | 2017-07-04 | Cross-network structured data collection system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107330089A CN107330089A (en) | 2017-11-07 |
| CN107330089B true CN107330089B (en) | 2020-06-23 |
Family
ID=60196289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710538423.2A Active CN107330089B (en) | 2017-07-04 | 2017-07-04 | Cross-network structured data collection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107330089B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061807A (en) * | 2019-11-23 | 2020-04-24 | 方正株式(武汉)科技开发有限公司 | Distributed data acquisition and analysis system and method, server and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101969475A (en) * | 2010-11-15 | 2011-02-09 | 张军 | Business data controllable distribution and fusion application system based on cloud computing |
| CN102006245A (en) * | 2010-12-10 | 2011-04-06 | 重庆亚德科技股份有限公司 | Date exchanging and sharing platform |
| CN105611650A (en) * | 2015-12-22 | 2016-05-25 | 河北远东通信系统工程有限公司 | Method for reducing coupling of application layer module and transfer layer module between multiple nodes |
| CN106657314A (en) * | 2016-12-14 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Cross-data center data synchronization system and method |
-
2017
- 2017-07-04 CN CN201710538423.2A patent/CN107330089B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101969475A (en) * | 2010-11-15 | 2011-02-09 | 张军 | Business data controllable distribution and fusion application system based on cloud computing |
| CN102006245A (en) * | 2010-12-10 | 2011-04-06 | 重庆亚德科技股份有限公司 | Date exchanging and sharing platform |
| CN105611650A (en) * | 2015-12-22 | 2016-05-25 | 河北远东通信系统工程有限公司 | Method for reducing coupling of application layer module and transfer layer module between multiple nodes |
| CN106657314A (en) * | 2016-12-14 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Cross-data center data synchronization system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107330089A (en) | 2017-11-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230144450A1 (en) | Multi-partitioning data for combination operations | |
| CN107729366B (en) | Universal multi-source heterogeneous large-scale data synchronization system | |
| CN103034735B (en) | Big data distributed file export method | |
| JP5818394B2 (en) | System and method for operating a mass data platform | |
| EP2954403B1 (en) | Cloud-based streaming data receiver and persister | |
| CN106682119B (en) | Asynchronous data synchronization method and system based on http service section and log system | |
| CN102880503B (en) | Data analysis system and data analysis method | |
| CN102262680B (en) | Distributed database proxy system based on massive data access requirement | |
| CN102375837B (en) | Data acquiring system and method | |
| CN104899274B (en) | A kind of memory database Efficient Remote access method | |
| KR20210005043A (en) | Input and output schema mapping | |
| US10552293B2 (en) | Logging as a service | |
| CN102385582A (en) | Method, server and system for processing production test data | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN111221793A (en) | Data mining method, platform, computer equipment and storage medium | |
| CN114401284A (en) | Real-time data acquisition and transmission system and method for fixed pollution source treatment working condition | |
| US9858302B1 (en) | Management of streaming data | |
| US11892976B2 (en) | Enhanced search performance using data model summaries stored in a remote data store | |
| CN116204239A (en) | Service processing method, device and computer readable storage medium | |
| CN113721856A (en) | Digital community management data storage system | |
| CN107330089B (en) | Cross-network structured data collection system | |
| CN102521339A (en) | System and method for dynamic access of data sources | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN115757642A (en) | Data synchronization method and device based on filing log file |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |