CN111143280B - Data scheduling method, system, device and storage medium - Google Patents
Data scheduling method, system, device and storage medium Download PDFInfo
- Publication number
- CN111143280B CN111143280B CN201911410145.8A CN201911410145A CN111143280B CN 111143280 B CN111143280 B CN 111143280B CN 201911410145 A CN201911410145 A CN 201911410145A CN 111143280 B CN111143280 B CN 111143280B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- database backup
- file
- distributing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012546 transfer Methods 0.000 claims abstract description 21
- 238000013515 script Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data scheduling method, a system, a device and a storage medium, wherein the method comprises the following steps: acquiring a database backup file and collecting the database backup file; distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data; the raw data is dumped to a data warehouse. The system comprises: the system comprises a collection module, a distribution module and a transfer module. The apparatus includes a memory and a processor for performing the above-described data scheduling method. By using the application, the data transfer speed can be improved. The method, the system, the device and the storage medium for data scheduling can be widely applied to the field of data transfer.
Description
Technical Field
The present application relates to the field of data transfer, and in particular, to a data scheduling method, system, device, and storage medium.
Background
In the current transfer mode, because centralized archiving and management are not performed on backup files of the traditional database, the backup files are required to be distributed to servers where corresponding databases are located before restoration, and data transmission also needs to be performed by means of a third-party tool. The transfer mode has the advantages of more links, low efficiency, large repeated workload, high error rate and high requirement for enterprise development and project survival, and needs a large amount of labor.
Disclosure of Invention
In order to solve the above technical problems, the present application is to provide a data scheduling method, system, device and storage medium, which can quickly transfer data to a designated data warehouse.
The first technical scheme adopted by the application is as follows: a data scheduling method, comprising the steps of:
acquiring a database backup file and collecting the database backup file;
distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data;
the original data are transferred to a data warehouse;
further, the step of obtaining the database backup file and collecting the database backup file specifically includes:
acquiring database data and generating a data file catalog;
and acquiring the needed database backup files from the data file catalogue, classifying the data files in a gathering way, and checking the file format after the classification of the data files in the gathering way is finished, so that the database backup files after the gathering are successfully obtained.
Further, the step of distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files to original data specifically includes:
distributing the database backup file to a server where a corresponding database is located based on the Netty framework;
calling a restore script to restore the database backup file and analyzing the restored data;
further, the step of transferring the original data to a data warehouse specifically includes:
reading a database based on the DataX, and acquiring original data in the database;
classifying the original data according to preset rules and transferring the classified data to a data warehouse;
further, the step of distributing the database backup file to the server where the corresponding database is located based on the Netty framework specifically includes:
establishing connection between a data center end and a distribution service center end;
and responding to a distribution request sent by the data center end, and distributing the database backup file to a corresponding server.
Further, the step of reading the database based on the DataX and obtaining the original data in the database specifically includes:
dividing the database into areas with different addresses according to the physical addresses of the database;
and creating a corresponding number of processing threads to scan each region and acquire data, and integrating to obtain the original data in the database.
Further, the response to the distribution request sent by the data center end adopts Protobuf serialization protocol to respond and send the request.
The second technical scheme adopted by the application is as follows: a data scheduling system, comprising:
the collection module is used for acquiring the database backup files and collecting the database backup files;
the distribution module is used for distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data;
and the transfer module is used for transferring the original data to the data warehouse.
The third technical scheme adopted by the application is as follows: a data scheduling apparatus comprising:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement the one data scheduling method as described above.
The fourth technical scheme adopted by the application is as follows: a storage medium having stored therein instructions executable by a processor, characterized by: the processor-executable instructions, when executed by the processor, are for implementing a data scheduling method as described above.
The method, the system, the device and the storage medium have the beneficial effects that: the unified scheduling of mass data is realized by collecting, distributing, restoring and restoring the database files, a data flow closed loop is formed, the speed of data acquisition and restoring is effectively improved, and the workload of data management staff is reduced.
Drawings
FIG. 1 is a flow chart of steps of a data scheduling method of the present application;
fig. 2 is a block diagram of a data scheduling system according to the present application.
Detailed Description
The application will now be described in further detail with reference to the drawings and to specific examples. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.
In some scenes needing to transfer data from a database and store the data, such as enterprises transferring the obtained data to an analysis platform, by using the method, the data transfer can be completed without a third-party tool, the data transfer speed is improved, the backup files are collected and classified through a data center end, the files are subjected to format verification, a complete data system is constructed, the data flow is effectively closed, and the workload of data staff is reduced.
As shown in fig. 1, the present application provides a data scheduling method, which includes the following steps:
s101, acquiring a database backup file and collecting the database backup file.
Specifically, the data information of each database can be manually added, the data information comprises structured data and unstructured data reported by each unit, and a user can visually add, delete, modify and review the files through a data center terminal.
S102, distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data;
s103, the original data are transferred to a data warehouse.
Specifically, data management is established for the restored original database, and in the step of the application, the data is transferred from the database to the data warehouse, partial transfer or full-library transfer can be realized through scripts, so that the workload is further lightened, and different types of data can be distributed to different servers through server management and the databases and servers are uniformly managed.
Further as a preferred embodiment of the method, the step of obtaining the database backup file and collecting the database backup file specifically includes:
acquiring database data and generating a data file catalog;
and acquiring the needed database backup files from the data file catalogue, classifying the data files in a gathering way, and checking the file format after the classification of the data files in the gathering way is finished, so that the database backup files after the gathering are successfully obtained.
Specifically, the visualized unified management of the data files is realized through the data file catalogue.
Further as a preferred embodiment of the method, the step of distributing the collected database backup files to each server and restoring the collected database backup files to original data based on the Netty framework specifically includes:
distributing the database backup file to a server where a corresponding database is located based on the Netty framework;
and calling a restore script to restore the database backup file and analyzing the restored data.
Specifically, netty is an asynchronous event-driven network application framework, and has the characteristics of high concurrency, high performance, high reliability and expansion, a two-layer thread model and NIO multiplexing non-blocking technology are adopted as a Netty framework bottom layer, and a handler chain in the framework can be expanded, a customizable serialization protocol and a network communication protocol.
Further as a preferred embodiment of the method, the step of transferring the raw data to a data warehouse specifically includes:
reading a database based on the DataX, and acquiring original data in the database;
classifying the original data according to preset rules and transferring the classified data to a data warehouse.
Specifically, the data X is an off-line synchronization tool for heterogeneous data sources, and aims to realize stable and efficient data synchronization functions among various heterogeneous data sources including a relational database (MySQL, oracle and the like).
Further as a preferred embodiment of the method, the step of distributing the database backup file to the server where the corresponding database is located based on the Netty framework specifically includes:
establishing connection between a data center end and a distribution server end;
and responding to a distribution request sent by the data center end, and distributing the database backup file to a corresponding server.
Specifically, the application comprises a data center end, a distribution service center end and a restoration service end in a server, wherein the data center end, the distribution service center end and the restoration service end in the server are respectively connected, the data center client end is used for managing the data of the whole system, the distribution service end is used for responding to a distribution request of the data center end to realize the data distribution to each server, the data center end sends a restoration request, and the restoration service end in the server responds to the request to restore the data file into original data.
Further as a preferred embodiment of the method, the step of reading the database based on DataX and obtaining the original data in the database specifically includes:
dividing the database into areas with different addresses according to the physical addresses of the database;
and creating a corresponding number of processing threads to scan each region and acquire data, and integrating to obtain the original data in the database.
Further as a preferred embodiment of the method, the response to the distribution request sent by the data center end adopts Protobuf serialization protocol to respond and send the request.
Specifically, the method for segmenting the database based on the physical address of the database, reducing the scanning range, extracting data at high speed, acquiring the original data of the database through multi-thread concurrent scanning, further improving the transfer speed, wherein Protobuf is a method for serializing structural data which is language independent, platform independent and extensible, can be used for (data) communication protocols and data storage, and can enable the serialization and deserialization speeds of a system to be higher by adopting the Protobuf serialization protocols.
Specific embodiments of the application are as follows:
the method comprises the steps of obtaining database backup files reported from each unit, integrating all database backup files to generate a data file catalog, classifying the collection of files required by a user and checking file formats, distributing the required database backup files to corresponding databases based on a Netty framework after the verification is successful, establishing connection between a data center end and a distribution server, sending a distribution request by the user through the data center end, responding the distribution request by the distribution server end, distributing the files to the corresponding servers, sending a restoration request by the data center end, calling a restoration script in the servers to restore the data files to obtain original data, and finally transferring the original data to the data warehouse through the dataX, wherein the user can realize monitoring and management of the whole data scheduling process through the data center end.
As shown in fig. 2, a data scheduling system includes:
the collection module is used for acquiring the database backup files and collecting the database backup files;
the distribution module is used for distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data;
and the transfer module is used for transferring the original data to the data warehouse.
Further as a preferred embodiment of the system, the collecting module further includes:
the catalog submodule is used for acquiring database data and generating a data file catalog;
and the classifying sub-module is used for acquiring the needed database backup files from the data file catalogue, classifying the collection, and checking the file format after the classification of the collection is finished.
Further as a preferred embodiment of the present system, the distribution module further comprises:
the server sub-module is used for distributing the database backup files to the servers where the corresponding databases are located based on the Netty framework;
the atom returning module is used for calling a restoring script to restore the database backup file and analyzing the restored data;
further as a preferred embodiment of the system, the transfer module further includes:
the reading sub-module is used for reading the database based on the DataX and acquiring the original data in the database;
and the classification sub-module is used for classifying the original data according to preset rules and transferring the classified data to a data warehouse.
The content in the method embodiment is applicable to the system embodiment, the functions specifically realized by the system embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method embodiment.
An authentication data scheduling apparatus:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement a data scheduling method as described above.
The content in the method embodiment is applicable to the embodiment of the device, and the functions specifically realized by the embodiment of the device are the same as those of the method embodiment, and the obtained beneficial effects are the same as those of the method embodiment.
A storage medium having stored therein instructions executable by a processor, characterized by: the processor-executable instructions, when executed by the processor, are for implementing a data scheduling method as described above.
The content in the method embodiment is applicable to the storage medium embodiment, and functions specifically implemented by the storage medium embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method embodiment.
While the preferred embodiment of the present application has been described in detail, the application is not limited to the embodiment, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the application, and these equivalent modifications and substitutions are intended to be included in the scope of the present application as defined in the appended claims.
Claims (9)
1. A method for scheduling data, comprising:
acquiring database data and generating a data file catalog;
acquiring a required database backup file from a data file catalog, classifying the required database backup file in a gathering way, and checking a file format after the classification of the gathering way is finished, so that the database backup file after the gathering is successfully obtained;
distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data;
the original data is transferred to a data warehouse, wherein partial transfer or full transfer can be performed through scripts.
2. The method for scheduling data according to claim 1, wherein the step of distributing the collected database backup files to each server based on the Netty framework and restoring the collected database backup files to the original data specifically comprises the following steps:
distributing the database backup file to a server where a corresponding database is located based on the Netty framework;
and calling a restore script to restore the database backup file and analyzing the restored data.
3. The method for scheduling data according to claim 1, wherein the step of transferring the original data to the data warehouse comprises:
reading a database based on the DataX, and acquiring original data in the database;
classifying the original data according to preset rules and transferring the classified data to a data warehouse.
4. The method for scheduling data according to claim 2, wherein the step of distributing the database backup file to the server where the corresponding database is located based on the Netty framework specifically comprises:
establishing connection between a data center end and a distribution server end;
and responding to a distribution request sent by the data center end, and distributing the database backup file to a corresponding server.
5. A data scheduling method according to claim 3, wherein the step of reading the database based on DataX and obtaining the original data in the database comprises:
dividing the database into areas with different addresses according to the physical addresses of the database;
and creating a corresponding number of processing threads to scan each region and acquire data, and integrating to obtain the original data in the database.
6. The method of claim 4, wherein the distributing request sent by the data center side specifically uses Protobuf serialization protocol to respond and send the request.
7. A data scheduling system, comprising:
the collection module is used for acquiring database data and generating a data file catalog; acquiring a required database backup file from a data file catalog, classifying the required database backup file in a gathering way, and checking a file format after the classification of the gathering way is finished, so that the database backup file after the gathering is successfully obtained;
the distribution module is used for distributing the collected database backup files to each server based on the Netty framework and restoring the database backup files into original data;
and the transfer module is used for transferring the original data to the data warehouse, wherein partial transfer or total transfer can be performed through the script.
8. A data scheduling apparatus, characterized by further comprising:
at least one processor;
at least one memory for storing at least one program;
when the at least one program is executed by the at least one processor, the at least one processor is caused to implement a data scheduling method as claimed in any one of claims 1-7.
9. A storage medium having stored therein instructions executable by a processor, characterized by: the processor executable instructions when executed by a processor are for implementing a data scheduling method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911410145.8A CN111143280B (en) | 2019-12-31 | 2019-12-31 | Data scheduling method, system, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911410145.8A CN111143280B (en) | 2019-12-31 | 2019-12-31 | Data scheduling method, system, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143280A CN111143280A (en) | 2020-05-12 |
| CN111143280B true CN111143280B (en) | 2023-10-10 |
Family
ID=70522493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911410145.8A Active CN111143280B (en) | 2019-12-31 | 2019-12-31 | Data scheduling method, system, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143280B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06236351A (en) * | 1993-02-10 | 1994-08-23 | Hitachi Ltd | Backup restoring device of on-line system |
| US5890165A (en) * | 1996-03-29 | 1999-03-30 | Emc Corporation | Method and apparatus for automatic discovery of databases |
| CN103605585A (en) * | 2013-11-06 | 2014-02-26 | 上海爱数软件有限公司 | Intelligent backup method based on data discovery |
| CN104317800A (en) * | 2014-09-19 | 2015-01-28 | 山东大学 | Hybrid storage system and method for mass intelligent power utilization data |
| CN104408064A (en) * | 2014-10-29 | 2015-03-11 | 中国建设银行股份有限公司 | Data storage method and device |
| CN105049874A (en) * | 2015-07-15 | 2015-11-11 | 华数传媒网络有限公司 | Integrated broadcast control platform and integrated broadcast control method |
| WO2015199345A1 (en) * | 2014-06-25 | 2015-12-30 | 주식회사 엔트리브 | Real-time data backup system and backup method using dual-emulator |
| CN105634799A (en) * | 2015-12-25 | 2016-06-01 | 国网浙江省电力公司湖州供电公司 | Diversified brand network equipment configuration automatic backup method |
| US9459969B1 (en) * | 2012-09-28 | 2016-10-04 | Emc Corporation | Method and system for enhanced backup database indexing |
| CN106648980A (en) * | 2016-12-12 | 2017-05-10 | 北京雷石天地电子技术有限公司 | Backup and recovery method and backup and recovery device for high-availability server databases in offline places |
| CN107239369A (en) * | 2017-04-19 | 2017-10-10 | 中国建设银行股份有限公司 | A kind of database filing standby system and method |
| CN108512785A (en) * | 2018-04-13 | 2018-09-07 | 南京优米亚信息科技有限公司 | A kind of Data Transport Protocol method |
| CN109254998A (en) * | 2018-09-04 | 2019-01-22 | 中国联合网络通信集团有限公司 | Data managing method, internet of things equipment, database server and system |
| CN109492040A (en) * | 2018-11-06 | 2019-03-19 | 深圳航天智慧城市系统技术研究院有限公司 | A kind of system suitable for data center's magnanimity short message data processing |
| CN110008704A (en) * | 2019-04-10 | 2019-07-12 | 河南城建学院 | A kind of electronic information intelligent memory system for industrial management |
| WO2019186941A1 (en) * | 2018-03-29 | 2019-10-03 | 株式会社島津製作所 | Analysis information collection program and analysis information collection system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8626741B2 (en) * | 2007-06-15 | 2014-01-07 | Emc Corporation | Process for cataloging data objects backed up from a content addressed storage system |
| KR20170133866A (en) * | 2016-05-27 | 2017-12-06 | 삼성에스디에스 주식회사 | Apparatus and method for data migration |
-
2019
- 2019-12-31 CN CN201911410145.8A patent/CN111143280B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06236351A (en) * | 1993-02-10 | 1994-08-23 | Hitachi Ltd | Backup restoring device of on-line system |
| US5890165A (en) * | 1996-03-29 | 1999-03-30 | Emc Corporation | Method and apparatus for automatic discovery of databases |
| US9459969B1 (en) * | 2012-09-28 | 2016-10-04 | Emc Corporation | Method and system for enhanced backup database indexing |
| CN103605585A (en) * | 2013-11-06 | 2014-02-26 | 上海爱数软件有限公司 | Intelligent backup method based on data discovery |
| WO2015199345A1 (en) * | 2014-06-25 | 2015-12-30 | 주식회사 엔트리브 | Real-time data backup system and backup method using dual-emulator |
| CN104317800A (en) * | 2014-09-19 | 2015-01-28 | 山东大学 | Hybrid storage system and method for mass intelligent power utilization data |
| CN104408064A (en) * | 2014-10-29 | 2015-03-11 | 中国建设银行股份有限公司 | Data storage method and device |
| CN105049874A (en) * | 2015-07-15 | 2015-11-11 | 华数传媒网络有限公司 | Integrated broadcast control platform and integrated broadcast control method |
| CN105634799A (en) * | 2015-12-25 | 2016-06-01 | 国网浙江省电力公司湖州供电公司 | Diversified brand network equipment configuration automatic backup method |
| CN106648980A (en) * | 2016-12-12 | 2017-05-10 | 北京雷石天地电子技术有限公司 | Backup and recovery method and backup and recovery device for high-availability server databases in offline places |
| CN107239369A (en) * | 2017-04-19 | 2017-10-10 | 中国建设银行股份有限公司 | A kind of database filing standby system and method |
| WO2019186941A1 (en) * | 2018-03-29 | 2019-10-03 | 株式会社島津製作所 | Analysis information collection program and analysis information collection system |
| CN108512785A (en) * | 2018-04-13 | 2018-09-07 | 南京优米亚信息科技有限公司 | A kind of Data Transport Protocol method |
| CN109254998A (en) * | 2018-09-04 | 2019-01-22 | 中国联合网络通信集团有限公司 | Data managing method, internet of things equipment, database server and system |
| CN109492040A (en) * | 2018-11-06 | 2019-03-19 | 深圳航天智慧城市系统技术研究院有限公司 | A kind of system suitable for data center's magnanimity short message data processing |
| CN110008704A (en) * | 2019-04-10 | 2019-07-12 | 河南城建学院 | A kind of electronic information intelligent memory system for industrial management |
Non-Patent Citations (2)
| Title |
|---|
| 于天等.一种高性能异步RPC框架的设计与实现.信息通信.2018,(2018年第03期),第127-129页. * |
| 梁丽琴 ; 郑少明 ; 郑汉军 ; 罗佳 ; .利用大数据技术进行海量数据治理.网络安全技术与应用.2018,(2018年第11期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143280A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8161085B2 (en) | Automatic and dynamic provisioning of databases | |
| CN108920153B (en) | Docker container dynamic scheduling method based on load prediction | |
| CN107220271B (en) | Method and system for storage processing and management of distributed digital resources | |
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| CN112732756A (en) | Data query method, device, equipment and storage medium | |
| CN116501783A (en) | Distributed database data importing method and system | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN111143280B (en) | Data scheduling method, system, device and storage medium | |
| CN112199432A (en) | High-performance data ETL device based on distribution and control method | |
| CN116186082A (en) | Data summarizing method based on distribution, first server and electronic equipment | |
| CN112667393B (en) | Method and device for building distributed task computing scheduling framework and computer equipment | |
| CN115022402A (en) | Agent acquisition method and system based on one-stack integration technology | |
| CN104735097A (en) | Information collecting method and system | |
| CN115426216A (en) | Internet of things node interconnection method based on complex virtual instrument software | |
| CN107330089B (en) | Cross-network structured data collection system | |
| CN106557530A (en) | Operation system, data recovery method and device | |
| CN116644039B (en) | Automatic acquisition and analysis method for online capacity operation log based on big data | |
| CN104391901B (en) | A kind of memory cell network big data base frame platform and its file access method | |
| CN117131000B (en) | NetCDF meteorological data processing method and terminal | |
| CN116991815B (en) | Log collection method, device, equipment and medium of distributed storage system | |
| CN113760836B (en) | Wide table calculation method and device | |
| CN118227572A (en) | Metadata change data reporting method and device, storage medium and electronic device | |
| CN116633752A (en) | Analysis management system based on big data | |
| CN115374058A (en) | Log data archiving method and device, storage medium and computer equipment | |
| CN114003410A (en) | Message-based flink real-time service scheduling configuration method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 510663 research institute office building, No.9, Kelin Road, Science City, Guangzhou high tech Industrial Development Zone, Guangzhou City, Guangdong Province Patentee after: GRG BANKING IT Co.,Ltd. Country or region after: China Patentee after: Guangdian Yuntong Group Co.,Ltd. Address before: 510663 research institute office building, No.9, Kelin Road, Science City, Guangzhou high tech Industrial Development Zone, Guangzhou City, Guangdong Province Patentee before: GRG BANKING IT Co.,Ltd. Country or region before: China Patentee before: GRG BANKING EQUIPMENT Co.,Ltd. |
|
| CP03 | Change of name, title or address |